Upload 15 files

README.md

@@ -1,12 +1,81 @@
----
-title: Gradio App
-emoji: 📊
-colorFrom: pink
-colorTo: blue
-sdk: gradio
-sdk_version: 3.44.3
-app_file: app.py
-pinned: false
----
-
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# Classroom Learning and Assessment Suite (CLAS)
+
+The Classroom Learning and Assessment Suite is a set of AI-driven solutions that complement in-class learning with interactive study and assessment. Primarily ideated by faculty interested in leveraging AI platforms to enhance student learning, the suite creates new mechanisms for both students and faculty members in both supporting high quality learning assessments and streamlining grading of student submissions using generative AI-based platforms.
+
+## Features
+CLAS is suitable for users of diverse programming backgrounds - from no programming background to seasoned programmers. The Suite provides:
+
+### **Prompting Guides**
+
+* **[A curated prompt dictionary for self-study](https://github.com/vanderbilt-data-science/lo-achievement/wiki/Guide-to-Learning-Objective-Prompts)**: A set of prompts that have been engineering and evaluated on OpenAI GPT 3.5 and GPT4 to provide numerous types of assessments for students engaging in self-study with instructor provided materials
+* **[A curated prompt dictionary for student assessment](https://github.com/vanderbilt-data-science/lo-achievement/wiki/Bloom's-Taxonomy-Rubric-Prompts)**: Prompts targeted primarily towards instructors who want a generalized rubric for evaluating student submissions
+
+### **Interface-Driven Programs**
+No programming experience? No problem. 
+
+* **[Student self-study using instructor assigned resources](https://huggingface.co/spaces/vanderbilt-dsi/selfstudy_learningobjectives_demo)**: A point-and-click interface with OpenAI generative AI. Upload your contextual coursework (book chapters, etc), and chat with your own customized tutor!
+* **[Hosted self- and in-class oral exam app]()**: Interested in using oral exams for assessment of student knowledge? Students - want to prepare for oral exams? Visit our point-and-click interface with OpenAI generative AI using your own coursework. You can upload questions provided by the instructor or ask the generative AI to assist in creating questions for you.
+
+### **Google Colab Notebooks**
+Want to evaluate students or want to customize your approach for interacting with generative AI?
+
+* **[Instructor Grading Notebook](https://github.com/vanderbilt-data-science/lo-achievement/blob/main/instructor_intr_notebook.ipynb)**: Upload a zip file of the JSON output of your students' exploration and assessment, and get insight on student strengths and weaknesses on the topic as well as structured feedback for all students.
+* **[Instructor Document Store Creation](https://github.com/vanderbilt-data-science/lo-achievement/blob/main/instructor_vector_store_creator.ipynb)**: Store all of your classroom content (pdfs, Youtube videos, website links) in a hosted location for easy accessibility for students working with generative AI platforms.
+* **[Prompting with Inline Context](https://github.com/vanderbilt-data-science/lo-achievement/blob/main/prompt_with_context.ipynb)**: A customizable, programmatic way to interface with generative AI using Google Colab through direct copy/paste of text content
+* **[Prompting with Vector Stores](https://github.com/vanderbilt-data-science/lo-achievement/blob/main/prompt_with_vector_store.ipynb)**: A customizable, programming way for self-study with generative AI using Google Colab through the creation of vector stores (better for larger corpuses of text)
+
+**Use the repo!**
+You can also directly clone/use the repo itself or use it as a package for development - this is great for experienced programmers or even those who would like to learn more about development with generative AI. The repo provides:
+- Generative AI and LangChain integration to process sources and create assessments and answer keys
+- Runs on Google Colab, with no additional installations needed
+You can also develop locally as desired and we encourage PR contributions from the community!
+
+## Getting Started
+
+There are a variety of ways to use CLAS:
+* **Navigate to the Wiki to explore prompts.** You can copy/paste/amend these in the interfaces provided by OpenAI, Google, Anthropic, etc.
+* **Use Google Colab to interact with notebooks.** Click on the notebook you'd like to enter in the files list above. You will see a blue Open In Colab link in the page that opens. Click this button to start your session in Google Colab, making sure that you're logged in with your Google Account.** It will take a few minutes to spin up and automatically install the required packages.
+* **Study through our hosted app.** Navigate to [CLAS on Huggingface](https://huggingface.co/spaces/vanderbilt-dsi/selfstudy_learningobjectives_demo). Follow the instructions to use the platform for self study.
+
+## Contributing
+
+To contribute to the project, please fork the repository and submit a pull request. Our community is supportive, and we provide training and classes if you're new to any of the frameworks used in the project. Everyone is welcome to contribute, as we believe participating in data science and AI projects is an excellent way to learn.
+
+## Community Guidelines
+
+We aim to create a welcoming and inclusive community where everyone can feel comfortable and valued, regardless of skill level, background, ability, or identity. To ensure a positive atmosphere, please adhere to our code of conduct and community guidelines.
+
+## Meetings
+
+- Sprint Planning & Retrospective: Mondays and Fridays at 10:30 am
+- Demos: Fridays at 3 pm
+
+## Additional Resources
+
+- LangChain documentation
+- Introduction to transformers and generative AI on our [YouTube channel](https://www.youtube.com/channel/UC8C2_3L5gR9qLmL7rmb2BdQ)
+- AI Summer and AI Winter sessions (free and open to all)
+
+## Reporting Issues
+
+If you encounter a bug, please submit an issue and label it with "Bug." To escalate the issue, email [datascience@vanderbilt.edu](mailto:datascience@vanderbilt.edu).
+
+## Contact Information
+
+- Organization: Data Science Institute at Vanderbilt University
+- Program: Data Science for Social Good
+- Main Email: [datascience@vanderbilt.edu](mailto:datascience@vanderbilt.edu)
+- Principal Investigators (PIs)
+  - Jesse Spencer-Smith, Ph.D., Chief Data Scientist, Data Science Institute, Vanderbilt University
+  - Jesse Blocher, Ph.D., Director of Graduate Studies, Data Science Institute, Vanderbilt University
+  - Dr. Yaa Kumah-Crystal, Ph.D., Pediatric Endocrinologist and Professor, Vanderbilt University Medical Center
+  - Charreau Bell, Ph.D., Senior Data Scientist, Data Science Institute, Vanderbilt University
+- Staff Lead: [charreau.s.bell@vanderbilt.edu](mailto:charreau.s.bell@vanderbilt.edu)  
+- Code Developers:
+  - Katrina Rbeiz, Ph.D. Student, Psychology, Vanderbilt University
+  - Minwoo Sohn, Graduate Student, Data Science, Vanderbilt University
+  - Ricky Sun, Graduate Student, Data Science, Vanderbilt University
+  - Eleanor Beers, Graduate Student, Data Science, Vanderbilt University
+  - Kevin Chen, Undergraduate, Computer Science, Vanderbilt University
+  - Adam Levav, Undergraduate, University of Maryland
+  - Varun Koduvayur, Undergraduate

UI_design_oral_exam_baseline_functionality.ipynb

@@ -0,0 +1,1454 @@
+{
+  "nbformat": 4,
+  "nbformat_minor": 0,
+  "metadata": {
+    "colab": {
+      "provenance": [],
+      "gpuType": "V100",
+      "include_colab_link": true
+    },
+    "kernelspec": {
+      "name": "python3",
+      "display_name": "Python 3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "view-in-github",
+        "colab_type": "text"
+      },
+      "source": [
+        "<a href=\"https://colab.research.google.com/github/vanderbilt-data-science/lo-achievement/blob/124-implement-baseline-functionality-for-oral-exam-module/UI_design_oral_exam_baseline_functionality.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "# Project IO Achievement - UI Design (Oral Exam)"
+      ],
+      "metadata": {
+        "id": "PIbogPXyM0wr"
+      }
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "## Problem Definition\n",
+        "\n",
+        "The v1 functionality for the Oral Exam module requires the following:\n",
+        "\n",
+        "1. Upload or generation of questions: either the user should upload a set of questions or we should allow the model to generate the questions. The user should pick or it should be inherent if there is no upload of questions. Note that we must also allow for context to be uploaded (vector store, vector store link, specific documents)\n",
+        "2. The model should prompt the user with a question and pause.\n",
+        "The user should respond by audio.\n",
+        "3. This should continue on until some final point where the exam is over.\n",
+        "\n",
+        "Then:\n",
+        "\n",
+        "1. We should use Whisper to do the transcription, and\n",
+        "2. Send the transcription, questions, and context for GPT4 for evaluation\n",
+        "Return the evaluation.\n",
+        "3. This will primarily be work on a user interface."
+      ],
+      "metadata": {
+        "id": "x_Vp8SiKM4p1"
+      }
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "## Libraries\n",
+        "\n",
+        "This section will install and import some important libraries such as Langchain, openai, Gradio, and so on"
+      ],
+      "metadata": {
+        "id": "o_60X8H3NEne"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# install libraries here\n",
+        "# -q flag for \"quiet\" install\n",
+        "%%capture\n",
+        "!pip install -q langchain\n",
+        "!pip install -q openai\n",
+        "!pip install -q gradio\n",
+        "!pip install -q transformers\n",
+        "!pip install -q datasets\n",
+        "!pip install -q huggingsound\n",
+        "!pip install -q torchaudio\n",
+        "!pip install -q git+https://github.com/openai/whisper.git\n",
+        "!pip install -q docx\n",
+        "!pip install -q PyPDF2\n",
+        "!pip install python-docx"
+      ],
+      "metadata": {
+        "id": "pxcqXgg2aAN7"
+      },
+      "execution_count": 1,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 2,
+      "metadata": {
+        "id": "pEjM1tLsMZBq"
+      },
+      "outputs": [],
+      "source": [
+        "# import libraries here\n",
+        "from langchain.llms import OpenAI\n",
+        "from langchain.prompts import PromptTemplate\n",
+        "from langchain.document_loaders import TextLoader\n",
+        "from langchain.indexes import VectorstoreIndexCreator\n",
+        "from langchain import ConversationChain, LLMChain, PromptTemplate\n",
+        "from langchain.chat_models import ChatOpenAI\n",
+        "from langchain.memory import ConversationBufferWindowMemory\n",
+        "from langchain.prompts import ChatPromptTemplate\n",
+        "from langchain.text_splitter import CharacterTextSplitter\n",
+        "from langchain.embeddings import OpenAIEmbeddings\n",
+        "from langchain.schema import SystemMessage, HumanMessage, AIMessage\n",
+        "import openai\n",
+        "import os\n",
+        "from getpass import getpass\n",
+        "from IPython.display import display, Javascript, HTML\n",
+        "from google.colab.output import eval_js\n",
+        "from base64 import b64decode\n",
+        "import ipywidgets as widgets\n",
+        "from IPython.display import clear_output\n",
+        "import time\n",
+        "import requests\n",
+        "from transformers import WhisperProcessor, WhisperForConditionalGeneration\n",
+        "from datasets import load_dataset\n",
+        "# from torchaudio.transforms import Resample\n",
+        "import whisper\n",
+        "from huggingsound import SpeechRecognitionModel\n",
+        "import numpy as np\n",
+        "import torch\n",
+        "import librosa\n",
+        "from datasets import load_dataset\n",
+        "from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor\n",
+        "from jiwer import wer\n",
+        "import pandas as pd\n",
+        "from IPython.display import display, HTML\n",
+        "import gradio as gr\n",
+        "from transformers import pipeline\n",
+        "from docx import Document\n",
+        "import PyPDF2\n",
+        "from pydub import AudioSegment\n",
+        "import tempfile\n",
+        "import os\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "## API Keys\n",
+        "\n",
+        "Use these cells to load the API keys required for this notebook. The below code cell uses the `getpass` library."
+      ],
+      "metadata": {
+        "id": "03KLZGI_a5W5"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "openai_api_key = getpass()\n",
+        "os.environ[\"OPENAI_API_KEY\"] = openai_api_key\n",
+        "openai.api_key = openai_api_key"
+      ],
+      "metadata": {
+        "id": "5smcWj4DbFgy",
+        "outputId": "9a73707b-1a6a-4253-b7d8-181a82b1040f",
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        }
+      },
+      "execution_count": 3,
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "··········\n"
+          ]
+        }
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "## Prompt Design\n",
+        "\n",
+        "To be added"
+      ],
+      "metadata": {
+        "id": "pMo9x8u4AEV1"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "chat = ChatOpenAI(temperature=0.0, model_name='gpt-3.5-turbo-16k')\n",
+        "chat"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "UgnCZRMhADvo",
+        "outputId": "1bd6b84d-3ea8-49ba-8156-701f4155d69c"
+      },
+      "execution_count": 4,
+      "outputs": [
+        {
+          "output_type": "execute_result",
+          "data": {
+            "text/plain": [
+              "ChatOpenAI(cache=None, verbose=False, callbacks=None, callback_manager=None, tags=None, metadata=None, client=<class 'openai.api_resources.chat_completion.ChatCompletion'>, model_name='gpt-3.5-turbo-16k', temperature=0.0, model_kwargs={}, openai_api_key='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', openai_api_base='', openai_organization='', openai_proxy='', request_timeout=None, max_retries=6, streaming=False, n=1, max_tokens=None, tiktoken_model_name=None)"
+            ]
+          },
+          "metadata": {},
+          "execution_count": 4
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# This is what I used to test the function 'generate_questions'\n",
+        "template_string2 = \"\"\"\n",
+        "You are teacher, and you will be given a {context} that is related to the presentation topic.\n",
+        "\n",
+        "Please generate a questions based on the context above and transcript that student created . \\\n",
+        "\n",
+        "The audio file generated by student is shown below: {transcribed_text}. \\\n",
+        "\"\"\""
+      ],
+      "metadata": {
+        "id": "WmysQZAhKBli"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "prompt_template1 = ChatPromptTemplate.from_template(template_string2)"
+      ],
+      "metadata": {
+        "id": "oij6W5rwAaGb"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# prompt_template.messages[0].prompt\n",
+        "prompt_template1.messages[0].prompt.input_variables"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "C1YRmL46AaJA",
+        "outputId": "e850524a-3831-4113-c796-5a0ec8584569"
+      },
+      "execution_count": null,
+      "outputs": [
+        {
+          "output_type": "execute_result",
+          "data": {
+            "text/plain": [
+              "['context', 'transcribed_text']"
+            ]
+          },
+          "metadata": {},
+          "execution_count": 25
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# This template is used for testing the function 'ai_evaluate'\n",
+        "# Detailed evaluation metrics are to be added\n",
+        "template_string3 = \"\"\"\n",
+        "You are teacher, and you will be given a context that is related to the presentation topic. \\\n",
+        "Now, given {context}, evaluate the answer based on the accuracy\n",
+        "\n",
+        "The main answer generated by student is shown below: {transcribed_text}. \\\n",
+        "The questions are shown below: {questions}. \\\n",
+        "The questions answered by student is shown below: {transcribed_qa}. \\\n",
+        "\"\"\"\n",
+        "prompt_template2 = ChatPromptTemplate.from_template(template_string3)\n",
+        "prompt_template2.messages[0].prompt.input_variables\n"
+      ],
+      "metadata": {
+        "id": "141Cxa2MT-l7",
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "outputId": "0374d45d-a7f6-41e7-aed0-44c61681de21"
+      },
+      "execution_count": null,
+      "outputs": [
+        {
+          "output_type": "execute_result",
+          "data": {
+            "text/plain": [
+              "['context', 'questions', 'transcribed_qa', 'transcribed_text']"
+            ]
+          },
+          "metadata": {},
+          "execution_count": 7
+        }
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "## Integrate Prompts from LO project"
+      ],
+      "metadata": {
+        "id": "MJCHl1T2TPWC"
+      }
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "### Creating a Chain for Short Answer Generation"
+      ],
+      "metadata": {
+        "id": "IPTyUOl-WdiL"
+      }
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "In this example, the context would include the poem \"The Road Not Taken\" by Robert Frost"
+      ],
+      "metadata": {
+        "id": "203qBjZvmFK1"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# This is what I used to test the function 'generate_questions_v2'\n",
+        "template_string = \"\"\"\n",
+        "You are a world-class tutor helping students to perform better on oral and written exams though interactive experiences.\"\n",
+        "\n",
+        "The following text should be used as the basis for the instructions which follow: {context} \\\n",
+        "\n",
+        "The following is the guideline for generating the questiion: {pre_prompt}\n",
+        "\"\"\""
+      ],
+      "metadata": {
+        "id": "w1AjHwIoVnvw"
+      },
+      "execution_count": 5,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "prompt_template = ChatPromptTemplate.from_template(template_string)\n",
+        "prompt_template.messages[0].prompt.input_variables"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "39-lm5I-Wlep",
+        "outputId": "44b39930-5258-484b-8c7d-c36ff4b5dc1a"
+      },
+      "execution_count": 6,
+      "outputs": [
+        {
+          "output_type": "execute_result",
+          "data": {
+            "text/plain": [
+              "['context', 'pre_prompt']"
+            ]
+          },
+          "metadata": {},
+          "execution_count": 6
+        }
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "### Creating a Chain for AI Evaluation"
+      ],
+      "metadata": {
+        "id": "-Pfxkcdxh9nZ"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "template_evaluation = \"\"\"\n",
+        "Given the follwing {context} and the {transcript}, evaluate whether or not the student answered correctly on the {question}.\n",
+        "\"\"\""
+      ],
+      "metadata": {
+        "id": "u6pH1x-gWnFF"
+      },
+      "execution_count": 7,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# @title\n",
+        "prompt_template2 = ChatPromptTemplate.from_template(template_evaluation)\n",
+        "prompt_template2.messages[0].prompt.input_variables"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "YPO_IE5ThC6W",
+        "outputId": "5361929c-cf8c-483d-901a-ed14a0db89fa"
+      },
+      "execution_count": 8,
+      "outputs": [
+        {
+          "output_type": "execute_result",
+          "data": {
+            "text/plain": [
+              "['context', 'question', 'transcript']"
+            ]
+          },
+          "metadata": {},
+          "execution_count": 8
+        }
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "## UI Design\n",
+        "\n",
+        "https://colab.research.google.com/github/petewarden/openai-whisper-webapp/blob/main/OpenAI_Whisper_ASR_Demo.ipynb"
+      ],
+      "metadata": {
+        "id": "M6IzVTjz5cex"
+      }
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "### Functions"
+      ],
+      "metadata": {
+        "id": "l4o8R5eUE1n8"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "def embed_key(openai_api_key):\n",
+        "  os.environ[\"OPENAI_API_KEY\"] = openai_api_key\n",
+        "\n",
+        "def transcribe(audio_file_path):\n",
+        "    with open(audio_file_path, \"rb\") as audio_file:\n",
+        "        # Call OpenAI's Whisper model for transcription\n",
+        "        transcript = openai.Audio.transcribe(\"whisper-1\", audio_file)\n",
+        "        transcribed_text = transcript[\"text\"]\n",
+        "        return transcribed_text\n",
+        "\n",
+        "def translate(text):\n",
+        "    # Create a prompt template (This will be changed later to fit the actual task)\n",
+        "    # Here translation will be a filler task of GPT\n",
+        "    test_input1 = prompt_template.format_messages(\n",
+        "              expertise='Language Translation',\n",
+        "              language='Japanese',\n",
+        "              style='romantic',\n",
+        "              transcribed_text=text)\n",
+        "\n",
+        "    response = chat.predict_messages(test_input1)\n",
+        "    return response.content\n",
+        "\n",
+        "def process_file(files):\n",
+        "  for file in files:\n",
+        "    try:\n",
+        "        extension = file.name.split('.')[-1].lower()\n",
+        "        if extension == 'docx':\n",
+        "            doc = Document(file.name)\n",
+        "            full_text = []\n",
+        "            for paragraph in doc.paragraphs:\n",
+        "                full_text.append(paragraph.text)\n",
+        "            return '\\n'.join(full_text)\n",
+        "\n",
+        "        elif extension == 'pdf':\n",
+        "            pdf_file = open(file.name, 'rb')\n",
+        "            reader = PyPDF2.PdfReader(pdf_file)\n",
+        "            num_pages = len(reader.pages)\n",
+        "            full_text = []\n",
+        "            for page in range(num_pages):\n",
+        "                page_obj = reader.pages[page]\n",
+        "                full_text.append(page_obj.extract_text())\n",
+        "            pdf_file.close()\n",
+        "            return '\\n'.join(full_text)\n",
+        "\n",
+        "        elif extension == 'txt':\n",
+        "            with open(file.name, 'r') as txt_file:\n",
+        "                full_text = txt_file.read()\n",
+        "            return full_text\n",
+        "\n",
+        "        else:\n",
+        "            return \"Unsupported file type\"\n",
+        "    except FileNotFoundError:\n",
+        "        return \"File not found\"\n",
+        "    except PermissionError:\n",
+        "        return \"Permission denied\"\n",
+        "\n",
+        "def generate_questions(context, transcript):\n",
+        "    text = process_file(context)\n",
+        "    test_input1 = prompt_template1.format_messages(\n",
+        "                      context = text,\n",
+        "                      transcribed_text = transcript)\n",
+        "\n",
+        "    response = chat(test_input1)\n",
+        "    return response.content\n",
+        "\n",
+        "def generate_questions_v2(text, prompt):\n",
+        "    #text = process_file(file)\n",
+        "    test_input1 = prompt_template.format_messages(\n",
+        "                      context = text,\n",
+        "                      pre_prompt = prompt)\n",
+        "\n",
+        "    response = chat(test_input1)\n",
+        "    return response\n",
+        "\n",
+        "# def ai_evaluate(context, audio_main, audio_qa, questions):\n",
+        "#     test_input1 = prompt_template2.format_messages(\n",
+        "#                       context = context,\n",
+        "#                       transcribed_text = audio_main,\n",
+        "#                       transcribed_qa = audio_qa,\n",
+        "#                       questions = questions)\n",
+        "\n",
+        "#     response = chat(test_input1)\n",
+        "#     return response.content\n",
+        "\n",
+        "def ai_evaluate_v2(text, audio_main, questions):\n",
+        "    #audio = transcribe(audio_main)\n",
+        "    test_input1 = prompt_template2.format_messages(\n",
+        "                      context = text,\n",
+        "                      transcript = audio_main,\n",
+        "                      question = questions\n",
+        "                      )\n",
+        "\n",
+        "    response = chat(test_input1)\n",
+        "    return response.content\n",
+        "\n",
+        "def upload_file(files):\n",
+        "    file_paths = [file.name for file in files]\n",
+        "    return file_paths"
+      ],
+      "metadata": {
+        "id": "ABN0X9xQHeii"
+      },
+      "execution_count": 12,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "### Test process_file"
+      ],
+      "metadata": {
+        "id": "a3WUL_hFyMkr"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "from google.colab import files\n",
+        "def upload_syllabi():\n",
+        "  uploaded = files.upload()\n",
+        "  for name, data in uploaded.items():\n",
+        "    with open(name, 'wb') as f:\n",
+        "      f.write(data)\n",
+        "    print('saved file', name)\n",
+        "upload_syllabi()"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/",
+          "height": 90
+        },
+        "id": "nih4FXX0Pl9U",
+        "outputId": "ce48c70a-d52c-4267-f3fc-8b22404448d7"
+      },
+      "execution_count": 13,
+      "outputs": [
+        {
+          "output_type": "display_data",
+          "data": {
+            "text/plain": [
+              "<IPython.core.display.HTML object>"
+            ],
+            "text/html": [
+              "\n",
+              "     <input type=\"file\" id=\"files-c72e8e8a-ac6f-48ab-9fb4-a84b1483268a\" name=\"files[]\" multiple disabled\n",
+              "        style=\"border:none\" />\n",
+              "     <output id=\"result-c72e8e8a-ac6f-48ab-9fb4-a84b1483268a\">\n",
+              "      Upload widget is only available when the cell has been executed in the\n",
+              "      current browser session. Please rerun this cell to enable.\n",
+              "      </output>\n",
+              "      <script>// Copyright 2017 Google LLC\n",
+              "//\n",
+              "// Licensed under the Apache License, Version 2.0 (the \"License\");\n",
+              "// you may not use this file except in compliance with the License.\n",
+              "// You may obtain a copy of the License at\n",
+              "//\n",
+              "//      http://www.apache.org/licenses/LICENSE-2.0\n",
+              "//\n",
+              "// Unless required by applicable law or agreed to in writing, software\n",
+              "// distributed under the License is distributed on an \"AS IS\" BASIS,\n",
+              "// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
+              "// See the License for the specific language governing permissions and\n",
+              "// limitations under the License.\n",
+              "\n",
+              "/**\n",
+              " * @fileoverview Helpers for google.colab Python module.\n",
+              " */\n",
+              "(function(scope) {\n",
+              "function span(text, styleAttributes = {}) {\n",
+              "  const element = document.createElement('span');\n",
+              "  element.textContent = text;\n",
+              "  for (const key of Object.keys(styleAttributes)) {\n",
+              "    element.style[key] = styleAttributes[key];\n",
+              "  }\n",
+              "  return element;\n",
+              "}\n",
+              "\n",
+              "// Max number of bytes which will be uploaded at a time.\n",
+              "const MAX_PAYLOAD_SIZE = 100 * 1024;\n",
+              "\n",
+              "function _uploadFiles(inputId, outputId) {\n",
+              "  const steps = uploadFilesStep(inputId, outputId);\n",
+              "  const outputElement = document.getElementById(outputId);\n",
+              "  // Cache steps on the outputElement to make it available for the next call\n",
+              "  // to uploadFilesContinue from Python.\n",
+              "  outputElement.steps = steps;\n",
+              "\n",
+              "  return _uploadFilesContinue(outputId);\n",
+              "}\n",
+              "\n",
+              "// This is roughly an async generator (not supported in the browser yet),\n",
+              "// where there are multiple asynchronous steps and the Python side is going\n",
+              "// to poll for completion of each step.\n",
+              "// This uses a Promise to block the python side on completion of each step,\n",
+              "// then passes the result of the previous step as the input to the next step.\n",
+              "function _uploadFilesContinue(outputId) {\n",
+              "  const outputElement = document.getElementById(outputId);\n",
+              "  const steps = outputElement.steps;\n",
+              "\n",
+              "  const next = steps.next(outputElement.lastPromiseValue);\n",
+              "  return Promise.resolve(next.value.promise).then((value) => {\n",
+              "    // Cache the last promise value to make it available to the next\n",
+              "    // step of the generator.\n",
+              "    outputElement.lastPromiseValue = value;\n",
+              "    return next.value.response;\n",
+              "  });\n",
+              "}\n",
+              "\n",
+              "/**\n",
+              " * Generator function which is called between each async step of the upload\n",
+              " * process.\n",
+              " * @param {string} inputId Element ID of the input file picker element.\n",
+              " * @param {string} outputId Element ID of the output display.\n",
+              " * @return {!Iterable<!Object>} Iterable of next steps.\n",
+              " */\n",
+              "function* uploadFilesStep(inputId, outputId) {\n",
+              "  const inputElement = document.getElementById(inputId);\n",
+              "  inputElement.disabled = false;\n",
+              "\n",
+              "  const outputElement = document.getElementById(outputId);\n",
+              "  outputElement.innerHTML = '';\n",
+              "\n",
+              "  const pickedPromise = new Promise((resolve) => {\n",
+              "    inputElement.addEventListener('change', (e) => {\n",
+              "      resolve(e.target.files);\n",
+              "    });\n",
+              "  });\n",
+              "\n",
+              "  const cancel = document.createElement('button');\n",
+              "  inputElement.parentElement.appendChild(cancel);\n",
+              "  cancel.textContent = 'Cancel upload';\n",
+              "  const cancelPromise = new Promise((resolve) => {\n",
+              "    cancel.onclick = () => {\n",
+              "      resolve(null);\n",
+              "    };\n",
+              "  });\n",
+              "\n",
+              "  // Wait for the user to pick the files.\n",
+              "  const files = yield {\n",
+              "    promise: Promise.race([pickedPromise, cancelPromise]),\n",
+              "    response: {\n",
+              "      action: 'starting',\n",
+              "    }\n",
+              "  };\n",
+              "\n",
+              "  cancel.remove();\n",
+              "\n",
+              "  // Disable the input element since further picks are not allowed.\n",
+              "  inputElement.disabled = true;\n",
+              "\n",
+              "  if (!files) {\n",
+              "    return {\n",
+              "      response: {\n",
+              "        action: 'complete',\n",
+              "      }\n",
+              "    };\n",
+              "  }\n",
+              "\n",
+              "  for (const file of files) {\n",
+              "    const li = document.createElement('li');\n",
+              "    li.append(span(file.name, {fontWeight: 'bold'}));\n",
+              "    li.append(span(\n",
+              "        `(${file.type || 'n/a'}) - ${file.size} bytes, ` +\n",
+              "        `last modified: ${\n",
+              "            file.lastModifiedDate ? file.lastModifiedDate.toLocaleDateString() :\n",
+              "                                    'n/a'} - `));\n",
+              "    const percent = span('0% done');\n",
+              "    li.appendChild(percent);\n",
+              "\n",
+              "    outputElement.appendChild(li);\n",
+              "\n",
+              "    const fileDataPromise = new Promise((resolve) => {\n",
+              "      const reader = new FileReader();\n",
+              "      reader.onload = (e) => {\n",
+              "        resolve(e.target.result);\n",
+              "      };\n",
+              "      reader.readAsArrayBuffer(file);\n",
+              "    });\n",
+              "    // Wait for the data to be ready.\n",
+              "    let fileData = yield {\n",
+              "      promise: fileDataPromise,\n",
+              "      response: {\n",
+              "        action: 'continue',\n",
+              "      }\n",
+              "    };\n",
+              "\n",
+              "    // Use a chunked sending to avoid message size limits. See b/62115660.\n",
+              "    let position = 0;\n",
+              "    do {\n",
+              "      const length = Math.min(fileData.byteLength - position, MAX_PAYLOAD_SIZE);\n",
+              "      const chunk = new Uint8Array(fileData, position, length);\n",
+              "      position += length;\n",
+              "\n",
+              "      const base64 = btoa(String.fromCharCode.apply(null, chunk));\n",
+              "      yield {\n",
+              "        response: {\n",
+              "          action: 'append',\n",
+              "          file: file.name,\n",
+              "          data: base64,\n",
+              "        },\n",
+              "      };\n",
+              "\n",
+              "      let percentDone = fileData.byteLength === 0 ?\n",
+              "          100 :\n",
+              "          Math.round((position / fileData.byteLength) * 100);\n",
+              "      percent.textContent = `${percentDone}% done`;\n",
+              "\n",
+              "    } while (position < fileData.byteLength);\n",
+              "  }\n",
+              "\n",
+              "  // All done.\n",
+              "  yield {\n",
+              "    response: {\n",
+              "      action: 'complete',\n",
+              "    }\n",
+              "  };\n",
+              "}\n",
+              "\n",
+              "scope.google = scope.google || {};\n",
+              "scope.google.colab = scope.google.colab || {};\n",
+              "scope.google.colab._files = {\n",
+              "  _uploadFiles,\n",
+              "  _uploadFilesContinue,\n",
+              "};\n",
+              "})(self);\n",
+              "</script> "
+            ]
+          },
+          "metadata": {}
+        },
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "Saving instructor_note_2.docx to instructor_note_2.docx\n",
+            "saved file instructor_note_2.docx\n"
+          ]
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Might need some way to make pdf file to load more readable\n",
+        "# process_file('/content/instrutor_note.docx')\n",
+        "# process_file('/content/Big Data & Economics.pdf')\n",
+        "# process_file('/content/Big Data & Economics.pdf')"
+      ],
+      "metadata": {
+        "id": "LJX1AKTMyVm8"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "### Gradio Interface V1"
+      ],
+      "metadata": {
+        "id": "c4s5o8baE6wN"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# @title\n",
+        "# with gr.Blocks() as demo:\n",
+        "#   gr.Markdown(\"# Oral Exam App\")\n",
+        "#   with gr.Box():\n",
+        "#     gr.HTML(\"\"\"Embed your OpenAI API key below; if you haven't created one already, visit\n",
+        "#       platform.openai.com/account/api-keys\n",
+        "#     to sign up for an account and get your personal API key\"\"\",\n",
+        "#             elem_classes=\"textbox_label\")\n",
+        "#     input = gr.Textbox(show_label=False, type=\"password\", container=False,\n",
+        "#                       placeholder=\"●●●●●●●●●●●●●●●●●\")\n",
+        "#     input.change(fn=embed_key, inputs=input, outputs=None)\n",
+        "\n",
+        "#   with gr.Blocks():\n",
+        "#       gr.Markdown(\"## Upload your audio file or start recording\")\n",
+        "\n",
+        "#       with gr.Row():\n",
+        "\n",
+        "\n",
+        "#         with gr.Column():\n",
+        "#           file_input = gr.Files(label=\"Load a mp3 file\",\n",
+        "#                               file_types=['.mp3'], type=\"file\",\n",
+        "#                               elem_classes=\"short-height\")\n",
+        "#           record_inputs = gr.Audio(source=\"microphone\", type=\"filepath\")\n",
+        "\n",
+        "#         with gr.Column():\n",
+        "#           outputs_transcribe=gr.Textbox(label=\"Transcription\")\n",
+        "\n",
+        "#           with gr.Row():\n",
+        "#             btn1 = gr.Button(value=\"Transcribe recorded audio\")\n",
+        "#             btn1.click(transcribe, inputs=record_inputs, outputs=outputs_transcribe)\n",
+        "#             btn2 = gr.Button(value=\"Transcribe uploaded audio\")\n",
+        "#             btn2.click(transcribe, inputs=file_input, outputs=outputs_transcribe)\n",
+        "\n",
+        "#           outputs_translate=gr.Textbox(label=\"Translation\")\n",
+        "#           btn3 = gr.Button(value=\"Translate\")\n",
+        "#           btn3.click(translate, inputs=outputs_transcribe, outputs=outputs_translate)\n",
+        "\n",
+        "#   demo.launch()\n"
+      ],
+      "metadata": {
+        "id": "ZkfJXCGDFhdw",
+        "cellView": "form"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "### baseline functionality V1"
+      ],
+      "metadata": {
+        "id": "AnkuosJ7Vw4z"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# @title\n",
+        "with gr.Blocks() as demo:\n",
+        "  gr.Markdown(\"# Oral Exam App\")\n",
+        "  gr.Markdown(\"## OpenAI API key\")\n",
+        "  with gr.Box():\n",
+        "    gr.HTML(\"\"\"Embed your OpenAI API key below; if you haven't created one already, visit\n",
+        "      platform.openai.com/account/api-keys\n",
+        "    to sign up for an account and get your personal API key\"\"\",\n",
+        "            elem_classes=\"textbox_label\")\n",
+        "    input = gr.Textbox(show_label=False, type=\"password\", container=False,\n",
+        "                      placeholder=\"●●●●●●●●●●●●●●●●●\")\n",
+        "    input.change(fn=embed_key, inputs=input, outputs=None)\n",
+        "\n",
+        "  with gr.Blocks():\n",
+        "  #########################\n",
+        "  #########Context#########\n",
+        "  #########################\n",
+        "      with gr.Accordion(\"Context section\"):\n",
+        "      ### Should also allow vector stores\n",
+        "          gr.Markdown(\"## Please upload the context document(s) for Oral exam\")\n",
+        "          context_input = gr.File(label=\"Click to upload context file\",\n",
+        "                                  file_count=\"multiple\",\n",
+        "                                  file_types=[\".txt\", \".docx\", \".pdf\"])\n",
+        "          outputs_context=gr.Textbox(label=\"Context\")\n",
+        "          context_input.change(fn=process_file, inputs=context_input, outputs=outputs_context)\n",
+        "          # upload_button = gr.Button(value=\"Show context\")\n",
+        "          # upload_button.click(process_file, context_input, outputs_context)\n",
+        "\n",
+        "  #########################\n",
+        "  #######Main Audio########\n",
+        "  #########################\n",
+        "      with gr.Accordion(\"Main audio section\"):\n",
+        "        gr.Markdown(\"## Upload your audio file or start recording\")\n",
+        "        with gr.Column():\n",
+        "          ## uploading files seem not working (don't know why)\n",
+        "          with gr.Row():\n",
+        "              file_input = gr.Audio(label=\"Upload Audio\", source=\"upload\", type=\"filepath\")\n",
+        "              record_inputs = gr.Audio(label=\"Record Audio\", source=\"microphone\", type=\"filepath\")\n",
+        "\n",
+        "          gr.Markdown(\"## Transcribe the audio uploaded or recorded\")\n",
+        "          outputs_transcribe=gr.Textbox(label=\"Transcription\")\n",
+        "\n",
+        "          file_input.change(fn=transcribe, inputs=file_input, outputs=outputs_transcribe)\n",
+        "          record_inputs.change(fn=transcribe, inputs=record_inputs, outputs=outputs_transcribe)\n",
+        "\n",
+        "  #########################\n",
+        "  ###Question Generation###\n",
+        "  #########################\n",
+        "      with gr.Accordion(\"Question section\"):\n",
+        "          gr.Markdown(\"## Questions\")\n",
+        "          with gr.Row():\n",
+        "            with gr.Column():\n",
+        "              outputs_qa=gr.Textbox(label=\"Generate questions\")\n",
+        "              btn3 = gr.Button(value=\"Generate questions\")\n",
+        "              btn3.click(generate_questions, inputs=[context_input, outputs_transcribe], outputs=outputs_qa)\n",
+        "\n",
+        "  ######################### Need additional work to include these questions when click button   #########################\n",
+        "            with gr.Column():\n",
+        "              submit_question=gr.Textbox(label=\"Use existing questions\")\n",
+        "              btn4 = gr.Button(value=\"Use these questions\")\n",
+        "              # btn4.click(use_this_question, inputs=outputs_transcribe, outputs=None)\n",
+        "\n",
+        "  #########################\n",
+        "  #########Audio QA########\n",
+        "  #########################\n",
+        "      with gr.Accordion(\"Audio QA section\"):\n",
+        "          gr.Markdown(\"## Question answering\")\n",
+        "          ##### This may be iterative\n",
+        "          with gr.Row():\n",
+        "            file_input2 = gr.Audio(label=\"Upload Audio\", source=\"upload\", type=\"filepath\")\n",
+        "            record_inputs2 = gr.Audio(label=\"Record Audio\", source=\"microphone\", type=\"filepath\")\n",
+        "\n",
+        "          gr.Markdown(\"## Transcribe the audio uploaded or recorded\")\n",
+        "          outputs_transcribe2=gr.Textbox(label=\"Transcription\")\n",
+        "          file_input2.change(fn=transcribe, inputs=file_input2, outputs=outputs_transcribe2)\n",
+        "          record_inputs2.change(fn=transcribe, inputs=record_inputs2, outputs=outputs_transcribe2)\n",
+        "\n",
+        "  #########################\n",
+        "  #######Evaluation########\n",
+        "  #########################\n",
+        "      with gr.Accordion(\"Evaluation section\"):\n",
+        "          gr.Markdown(\"## Evaluation\")\n",
+        "          with gr.Tab(\"General evalution\"):\n",
+        "            evalution=gr.Textbox(label=\"AI Evaluation\")\n",
+        "            btn5 = gr.Button(value=\"Evaluate\")\n",
+        "            btn5.click(ai_evaluate, inputs=[context_input, record_inputs,record_inputs2, outputs_qa], outputs=evalution)\n",
+        "          with gr.Tab(\"Quantitative evalution\"):\n",
+        "            table_output = gr.Dataframe(label = \"Some kind of evaluation metrics?\")\n",
+        "            btn6 = gr.Button(value=\"Evaluate\")\n",
+        "            btn6.click(ai_evaluate, inputs=[context_input, record_inputs,record_inputs2, outputs_qa], outputs=table_output)\n",
+        "\n",
+        "  demo.launch()\n",
+        "  # demo.launch(share=True)\n",
+        "  # demo.launch(debug=True)"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/",
+          "height": 616
+        },
+        "id": "EAPljDMYVy3u",
+        "outputId": "1f347376-14e8-48ea-e531-295a4fefd6cd",
+        "cellView": "form"
+      },
+      "execution_count": null,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "Colab notebook detected. To show errors in colab notebook, set debug=True in launch()\n",
+            "Note: opening Chrome Inspector may crash demo inside Colab notebooks.\n",
+            "\n",
+            "To create a public link, set `share=True` in `launch()`.\n"
+          ]
+        },
+        {
+          "output_type": "display_data",
+          "data": {
+            "text/plain": [
+              "<IPython.core.display.Javascript object>"
+            ],
+            "application/javascript": [
+              "(async (port, path, width, height, cache, element) => {\n",
+              "                        if (!google.colab.kernel.accessAllowed && !cache) {\n",
+              "                            return;\n",
+              "                        }\n",
+              "                        element.appendChild(document.createTextNode(''));\n",
+              "                        const url = await google.colab.kernel.proxyPort(port, {cache});\n",
+              "\n",
+              "                        const external_link = document.createElement('div');\n",
+              "                        external_link.innerHTML = `\n",
+              "                            <div style=\"font-family: monospace; margin-bottom: 0.5rem\">\n",
+              "                                Running on <a href=${new URL(path, url).toString()} target=\"_blank\">\n",
+              "                                    https://localhost:${port}${path}\n",
+              "                                </a>\n",
+              "                            </div>\n",
+              "                        `;\n",
+              "                        element.appendChild(external_link);\n",
+              "\n",
+              "                        const iframe = document.createElement('iframe');\n",
+              "                        iframe.src = new URL(path, url).toString();\n",
+              "                        iframe.height = height;\n",
+              "                        iframe.allow = \"autoplay; camera; microphone; clipboard-read; clipboard-write;\"\n",
+              "                        iframe.width = width;\n",
+              "                        iframe.style.border = 0;\n",
+              "                        element.appendChild(iframe);\n",
+              "                    })(7862, \"/\", \"100%\", 500, false, window.element)"
+            ]
+          },
+          "metadata": {}
+        }
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "### Baseline Functionality V2"
+      ],
+      "metadata": {
+        "id": "YAKl-5P4dHEF"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# @title\n",
+        "with gr.Blocks() as demo:\n",
+        "  gr.Markdown(\"# Oral Exam App\")\n",
+        "  gr.Markdown(\"## OpenAI API key\")\n",
+        "  with gr.Box():\n",
+        "    gr.HTML(\"\"\"Embed your OpenAI API key below; if you haven't created one already, visit\n",
+        "      platform.openai.com/account/api-keys\n",
+        "    to sign up for an account and get your personal API key\"\"\",\n",
+        "            elem_classes=\"textbox_label\")\n",
+        "    input = gr.Textbox(show_label=False, type=\"password\", container=False,\n",
+        "                      placeholder=\"●●●●●●●●●●●●●●●●●\")\n",
+        "    input.change(fn=embed_key, inputs=input, outputs=None)\n",
+        "\n",
+        "  with gr.Blocks():\n",
+        "  #########################\n",
+        "  #########Context#########\n",
+        "  #########################\n",
+        "      with gr.Accordion(\"Context section\"):\n",
+        "      ### Should also allow vector stores\n",
+        "          gr.Markdown(\"## Please upload the context document(s) for Oral exam\")\n",
+        "          context_input = gr.File(label=\"Click to upload context file\",\n",
+        "                                  file_count=\"multiple\",\n",
+        "                                  file_types=[\".txt\", \".docx\", \".pdf\"])\n",
+        "          outputs_context=gr.Textbox(label=\"Context\")\n",
+        "          context_input.change(fn=process_file, inputs=context_input, outputs=outputs_context)\n",
+        "          # upload_button = gr.Button(value=\"Show context\")\n",
+        "          # upload_button.click(process_file, context_input, outputs_context)\n",
+        "\n",
+        "  #########################\n",
+        "  ###Question Generation###\n",
+        "  #########################\n",
+        "      with gr.Accordion(\"Question section\"):\n",
+        "          gr.Markdown(\"## Questions\")\n",
+        "          with gr.Row():\n",
+        "            with gr.Column():\n",
+        "              outputs_qa=gr.Textbox(label=\"Generate questions\")\n",
+        "              btn1 = gr.Button(value=\"Generate questions\")\n",
+        "              btn1.click(generate_questions_v2, inputs=outputs_context, outputs=outputs_qa)\n",
+        "\n",
+        "  ######################### Need additional work to include these questions when click button   #########################\n",
+        "            with gr.Column():\n",
+        "              submit_question=gr.Textbox(label=\"Use existing questions\")\n",
+        "              btn4 = gr.Button(value=\"Use these questions\")\n",
+        "              # btn4.click(use_this_question, inputs=outputs_transcribe, outputs=None)\n",
+        "\n",
+        "  #########################\n",
+        "  #######Main Audio########\n",
+        "  #########################\n",
+        "      with gr.Accordion(\"Main audio section\"):\n",
+        "        gr.Markdown(\"## Upload your audio file or start recording\")\n",
+        "        with gr.Column():\n",
+        "          ## uploading files seem not working (don't know why)\n",
+        "          with gr.Row():\n",
+        "              file_input = gr.Audio(label=\"Upload Audio\", source=\"upload\", type=\"filepath\")\n",
+        "              record_inputs = gr.Audio(label=\"Record Audio\", source=\"microphone\", type=\"filepath\")\n",
+        "\n",
+        "          gr.Markdown(\"## Transcribe the audio uploaded or recorded\")\n",
+        "          outputs_transcribe=gr.Textbox(label=\"Transcription\")\n",
+        "\n",
+        "          file_input.change(fn=transcribe, inputs=file_input, outputs=outputs_transcribe)\n",
+        "          record_inputs.change(fn=transcribe, inputs=record_inputs, outputs=outputs_transcribe)\n",
+        "\n",
+        "  #########################\n",
+        "  #######Evaluation########\n",
+        "  #########################\n",
+        "      with gr.Accordion(\"Evaluation section\"):\n",
+        "          gr.Markdown(\"## Evaluation\")\n",
+        "          with gr.Tab(\"General evalution\"):\n",
+        "            evalution=gr.Textbox(label=\"AI Evaluation\")\n",
+        "            btn5 = gr.Button(value=\"Evaluate\")\n",
+        "            btn5.click(ai_evaluate_v2, inputs=[outputs_context, outputs_transcribe, outputs_qa], outputs=evalution)\n",
+        "          with gr.Tab(\"Quantitative evalution\"):\n",
+        "            table_output = gr.Dataframe(label = \"Some kind of evaluation metrics?\")\n",
+        "            btn6 = gr.Button(value=\"Evaluate\")\n",
+        "            btn6.click(ai_evaluate_v2, inputs=[outputs_context, outputs_transcribe, outputs_qa], outputs=table_output)\n",
+        "\n",
+        "  demo.launch()"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/",
+          "height": 706
+        },
+        "id": "04KxUQgUcTrm",
+        "outputId": "66a9f8c8-36fe-4792-b7d6-3befa6f09269",
+        "cellView": "form",
+        "collapsed": true
+      },
+      "execution_count": 23,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stderr",
+          "text": [
+            "/usr/local/lib/python3.10/dist-packages/gradio/utils.py:833: UserWarning: Expected 2 arguments for function <function generate_questions_v2 at 0x7aa8748f9bd0>, received 1.\n",
+            "  warnings.warn(\n",
+            "/usr/local/lib/python3.10/dist-packages/gradio/utils.py:837: UserWarning: Expected at least 2 arguments for function <function generate_questions_v2 at 0x7aa8748f9bd0>, received 1.\n",
+            "  warnings.warn(\n"
+          ]
+        },
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "Colab notebook detected. To show errors in colab notebook, set debug=True in launch()\n",
+            "Note: opening Chrome Inspector may crash demo inside Colab notebooks.\n",
+            "\n",
+            "To create a public link, set `share=True` in `launch()`.\n"
+          ]
+        },
+        {
+          "output_type": "display_data",
+          "data": {
+            "text/plain": [
+              "<IPython.core.display.Javascript object>"
+            ],
+            "application/javascript": [
+              "(async (port, path, width, height, cache, element) => {\n",
+              "                        if (!google.colab.kernel.accessAllowed && !cache) {\n",
+              "                            return;\n",
+              "                        }\n",
+              "                        element.appendChild(document.createTextNode(''));\n",
+              "                        const url = await google.colab.kernel.proxyPort(port, {cache});\n",
+              "\n",
+              "                        const external_link = document.createElement('div');\n",
+              "                        external_link.innerHTML = `\n",
+              "                            <div style=\"font-family: monospace; margin-bottom: 0.5rem\">\n",
+              "                                Running on <a href=${new URL(path, url).toString()} target=\"_blank\">\n",
+              "                                    https://localhost:${port}${path}\n",
+              "                                </a>\n",
+              "                            </div>\n",
+              "                        `;\n",
+              "                        element.appendChild(external_link);\n",
+              "\n",
+              "                        const iframe = document.createElement('iframe');\n",
+              "                        iframe.src = new URL(path, url).toString();\n",
+              "                        iframe.height = height;\n",
+              "                        iframe.allow = \"autoplay; camera; microphone; clipboard-read; clipboard-write;\"\n",
+              "                        iframe.width = width;\n",
+              "                        iframe.style.border = 0;\n",
+              "                        element.appendChild(iframe);\n",
+              "                    })(7863, \"/\", \"100%\", 500, false, window.element)"
+            ]
+          },
+          "metadata": {}
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "def prompt_select(selection, number, length):\n",
+        "  if selection == \"Random\":\n",
+        "    prompt = f\"Please design a {number} question quiz based on the context provided and the inputted learning objectives (if applicable). The types of questions should be randomized (including multiple choice, short answer, true/false, short answer, etc.).  Provide one question at a time, and wait for my response before providing me with feedback. Again, while the quiz may ask for multiple questions, you should only provide 1 question in you initial response. Do not include the answer in your response. If I get an answer wrong, provide me with an explanation of why it was incorrect, and then give me additional chances to respond until I get the correct choice. Explain why the correct choice is right.\"\n",
+        "  elif selection == \"Fill in the Blank\":\n",
+        "    prompt = f\"Create a {number} question fill in the blank quiz refrencing the context provided. The quiz should reflect the learning objectives (if inputted). The 'blank' part of the question should appear as '________'. The answers should reflect what word(s) should go in the blank an accurate statement. An example is the follow: 'The author of the article is ______.' The question should be a statement. Provide one question at a time, and wait for my response before providing me with feedback. Again, while the quiz may ask for multiple questions, you should only provide ONE question in you initial response. Do not include the answer in your response. If I get an answer wrong, provide me with an explanation of why it was incorrect,and then give me additional chances to respond until I get the correct choice. Explain why the correct choice is right.\"\n",
+        "  elif selection == \"Short Answer\":\n",
+        "    prompt = f\"Please design a {number} question quiz about which reflects the learning objectives (if inputted).  The questions should be short answer. Expect the correct answers to be {length} sentences long. Provide one question at a time, and wait for my response before providing me with feedback. Again, while the quiz may ask for multiple questions, you should only provide ONE question in you initial response. Do not include the answer in your response. If I get an answer wrong, provide me with an explanation of why it was incorrect, and then give me additional chances to respond until I get the correct choice. Explain why the correct answer is right.\"\n",
+        "  else:\n",
+        "    prompt = f\"Please design a {number} question {selection.lower()} quiz based on the context provided and the inputted learning objectives (if applicable). Provide one question at a time, and wait for my response before providing me with feedback. Again, while the quiz may ask for multiple questions, you should only provide 1 question in you initial response. Do not include the answer in your response. If I get an answer wrong, provide me with an explanation of why it was incorrect, and then give me additional chances to respond until I get the correct choice. Explain why the correct choice is right.\"\n",
+        "  return prompt\n",
+        "\n",
+        "\n",
+        "# Function to save prompts (premade or custom) and return in the user input box in the chatbot`\n",
+        "saved_text = \"\"\n",
+        "def save_text(text):\n",
+        "    global saved_text\n",
+        "    saved_text = text\n",
+        "\n",
+        "def return_text():\n",
+        "    # Return the saved text\n",
+        "    return saved_text"
+      ],
+      "metadata": {
+        "id": "wF80F1wU80rU"
+      },
+      "execution_count": 14,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "### Baseline Functionality V3"
+      ],
+      "metadata": {
+        "id": "F5-Ja2evCE4X"
+      }
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Updated Question Selection and Chatbot Feature"
+      ],
+      "metadata": {
+        "id": "rr8YlzcJCKv4"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "with gr.Blocks() as demo:\n",
+        "  gr.Markdown(\"# Oral Exam App\")\n",
+        "  gr.Markdown(\"## OpenAI API key\")\n",
+        "  with gr.Box():\n",
+        "    gr.HTML(\"\"\"Embed your OpenAI API key below; if you haven't created one already, visit\n",
+        "      platform.openai.com/account/api-keys\n",
+        "    to sign up for an account and get your personal API key\"\"\",\n",
+        "            elem_classes=\"textbox_label\")\n",
+        "    input = gr.Textbox(show_label=False, type=\"password\", container=False,\n",
+        "                      placeholder=\"●●●●●●●●●●●●●●●●●\")\n",
+        "    input.change(fn=embed_key, inputs=input, outputs=None)\n",
+        "\n",
+        "  with gr.Blocks():\n",
+        "  #########################\n",
+        "  #########Context#########\n",
+        "  #########################\n",
+        "      with gr.Accordion(\"Context section\"):\n",
+        "      ### Should also allow vector stores\n",
+        "          gr.Markdown(\"## Please upload the context document(s) for Oral exam\")\n",
+        "          context_input = gr.File(label=\"Click to upload context file\",\n",
+        "                                  file_count=\"multiple\",\n",
+        "                                  file_types=[\".txt\", \".docx\", \".pdf\"])\n",
+        "          outputs_context=gr.Textbox(label=\"Context\")\n",
+        "          context_input.change(fn=process_file, inputs=context_input, outputs=outputs_context)\n",
+        "          # upload_button = gr.Button(value=\"Show context\")\n",
+        "          # upload_button.click(process_file, context_input, outputs_context)\n",
+        "\n",
+        "      with gr.Blocks():\n",
+        "        gr.Markdown(\"\"\"\n",
+        "        ## Generate a Premade Prompt\n",
+        "        Select your type and number of desired questions. Click \"Generate Prompt\" to get your premade prompt,\n",
+        "        and then \"Insert Prompt into Chat\" to copy the text into the chat interface below. \\\n",
+        "        You can also copy the prompt using the icon in the upper right corner and paste directly into the input box when interacting with the model.\n",
+        "        \"\"\")\n",
+        "        with gr.Row():\n",
+        "          with gr.Column():\n",
+        "            question_type = gr.Dropdown([\"Multiple Choice\", \"True or False\", \"Short Answer\", \"Fill in the Blank\", \"Random\"], label=\"Question Type\")\n",
+        "            number_of_questions = gr.Textbox(label=\"Enter desired number of questions\")\n",
+        "            sa_desired_length = gr.Dropdown([\"1-2\", \"3-4\", \"5-6\", \"6 or more\"], label = \"For short answer questions only, choose the desired sentence length for answers. The default value is 1-2 sentences.\")\n",
+        "          with gr.Column():\n",
+        "            prompt_button = gr.Button(\"Generate Prompt\")\n",
+        "            premade_prompt_output = gr.Textbox(label=\"Generated prompt (save or copy)\", show_copy_button=True)\n",
+        "            prompt_button.click(prompt_select,\n",
+        "                            inputs=[question_type, number_of_questions, sa_desired_length],\n",
+        "                            outputs=premade_prompt_output)\n",
+        "  ########################\n",
+        "  ##Question Generation###\n",
+        "  ########################\n",
+        "      with gr.Accordion(\"Question section\"):\n",
+        "          gr.Markdown(\"## Questions\")\n",
+        "          with gr.Row():\n",
+        "            with gr.Column():\n",
+        "              outputs_qa=gr.Textbox(label=\"Generate questions\")\n",
+        "              btn1 = gr.Button(value=\"Generate questions\")\n",
+        "              btn1.click(generate_questions_v2, inputs=[outputs_context, premade_prompt_output], outputs=outputs_qa)\n",
+        "\n",
+        "  ######################### Need additional work to include these questions when click button   #########################\n",
+        "            with gr.Column():\n",
+        "              submit_question=gr.Textbox(label=\"Use existing questions\")\n",
+        "              btn4 = gr.Button(value=\"Use these questions\")\n",
+        "              # btn4.click(use_this_question, inputs=outputs_transcribe, outputs=None)\n",
+        "\n",
+        "  #########################\n",
+        "  #######Main Audio########\n",
+        "  #########################\n",
+        "      with gr.Accordion(\"Main audio section\"):\n",
+        "        gr.Markdown(\"## Upload your audio file or start recording\")\n",
+        "        with gr.Column():\n",
+        "          ## uploading files seem not working (don't know why)\n",
+        "          with gr.Row():\n",
+        "              file_input = gr.Audio(label=\"Upload Audio\", source=\"upload\", type=\"filepath\")\n",
+        "              record_inputs = gr.Audio(label=\"Record Audio\", source=\"microphone\", type=\"filepath\")\n",
+        "\n",
+        "          gr.Markdown(\"## Transcribe the audio uploaded or recorded\")\n",
+        "          outputs_transcribe=gr.Textbox(label=\"Transcription\")\n",
+        "\n",
+        "          file_input.change(fn=transcribe, inputs=file_input, outputs=outputs_transcribe)\n",
+        "          record_inputs.change(fn=transcribe, inputs=record_inputs, outputs=outputs_transcribe)\n",
+        "\n",
+        "  #########################\n",
+        "  #######Evaluation########\n",
+        "  #########################\n",
+        "      with gr.Accordion(\"Evaluation section\"):\n",
+        "          gr.Markdown(\"## Evaluation\")\n",
+        "          with gr.Tab(\"General evalution\"):\n",
+        "            evalution=gr.Textbox(label=\"AI Evaluation\")\n",
+        "            btn5 = gr.Button(value=\"Evaluate\")\n",
+        "            btn5.click(ai_evaluate_v2, inputs=[outputs_context, outputs_transcribe, outputs_qa], outputs=evalution)\n",
+        "          with gr.Tab(\"Quantitative evalution\"):\n",
+        "            table_output = gr.Dataframe(label = \"Some kind of evaluation metrics?\")\n",
+        "            btn6 = gr.Button(value=\"Evaluate\")\n",
+        "            btn6.click(ai_evaluate_v2, inputs=[outputs_context, outputs_transcribe, outputs_qa], outputs=table_output)\n",
+        "\n",
+        "\n",
+        "      # Chatbot (https://gradio.app/creating-a-chatbot/)\n",
+        "      '''\n",
+        "      with gr.Blocks():\n",
+        "        gr.Markdown(\"\"\"\n",
+        "        ## Chat with the Model\n",
+        "        Click \"Display Prompt\" to display the premade or custom prompt that you created earlier. Then, continue chatting with the model.\n",
+        "        \"\"\")\n",
+        "        with gr.Row():\n",
+        "          show_prompt_block = gr.Button(\"Display Prompt\")\n",
+        "      '''\n",
+        "      gr.Markdown(\"## Chat with the Model\")\n",
+        "      with gr.Row(equal_height=True):\n",
+        "        with gr.Column(scale=2):\n",
+        "          chatbot = gr.Chatbot()\n",
+        "          with gr.Row():\n",
+        "            user_chat_input = gr.Textbox(label=\"User input\", scale=9)\n",
+        "            user_chat_input.submit(return_text, inputs=None, outputs=user_chat_input)\n",
+        "            user_chat_submit = gr.Button(\"Ask/answer model\", scale=1)\n",
+        "            #show_prompt_block.click(return_text, inputs=None, outputs=user_chat_input)\n",
+        "\n",
+        "        # TODO Move the sources so it's displayed to the right of the chat bot,\n",
+        "        # with the sources taking up about 1/3rd of the horizontal space\n",
+        "        # with gr.Box(elem_id=\"sources-container\", scale=1):\n",
+        "        #   # TODO: Display document sources in a nicer format?\n",
+        "        #   gr.HTML(value=\"<h3 id='sources'>Sources</h3>\")\n",
+        "        #   sources_output = []\n",
+        "        #   for i in range(num_sources):\n",
+        "        #     source_elem = gr.HTML(visible=False)\n",
+        "        #     sources_output.append(source_elem)\n",
+        "\n",
+        "demo.launch()"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/",
+          "height": 616
+        },
+        "id": "Y7-3JFuZ8H5k",
+        "outputId": "ea99ce65-7b79-4d39-dd88-44785b0d6615"
+      },
+      "execution_count": 24,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "Colab notebook detected. To show errors in colab notebook, set debug=True in launch()\n",
+            "Note: opening Chrome Inspector may crash demo inside Colab notebooks.\n",
+            "\n",
+            "To create a public link, set `share=True` in `launch()`.\n"
+          ]
+        },
+        {
+          "output_type": "display_data",
+          "data": {
+            "text/plain": [
+              "<IPython.core.display.Javascript object>"
+            ],
+            "application/javascript": [
+              "(async (port, path, width, height, cache, element) => {\n",
+              "                        if (!google.colab.kernel.accessAllowed && !cache) {\n",
+              "                            return;\n",
+              "                        }\n",
+              "                        element.appendChild(document.createTextNode(''));\n",
+              "                        const url = await google.colab.kernel.proxyPort(port, {cache});\n",
+              "\n",
+              "                        const external_link = document.createElement('div');\n",
+              "                        external_link.innerHTML = `\n",
+              "                            <div style=\"font-family: monospace; margin-bottom: 0.5rem\">\n",
+              "                                Running on <a href=${new URL(path, url).toString()} target=\"_blank\">\n",
+              "                                    https://localhost:${port}${path}\n",
+              "                                </a>\n",
+              "                            </div>\n",
+              "                        `;\n",
+              "                        element.appendChild(external_link);\n",
+              "\n",
+              "                        const iframe = document.createElement('iframe');\n",
+              "                        iframe.src = new URL(path, url).toString();\n",
+              "                        iframe.height = height;\n",
+              "                        iframe.allow = \"autoplay; camera; microphone; clipboard-read; clipboard-write;\"\n",
+              "                        iframe.width = width;\n",
+              "                        iframe.style.border = 0;\n",
+              "                        element.appendChild(iframe);\n",
+              "                    })(7864, \"/\", \"100%\", 500, false, window.element)"
+            ]
+          },
+          "metadata": {}
+        },
+        {
+          "output_type": "execute_result",
+          "data": {
+            "text/plain": []
+          },
+          "metadata": {},
+          "execution_count": 24
+        }
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "### What's left\n",
+        "- vector store (link) upload\n",
+        "- submit question section need to be linked with ai_evaluate function"
+      ],
+      "metadata": {
+        "id": "g2EVIogW69Fd"
+      }
+    }
+  ]
+}

UI_design_oral_exam_chatbot.ipynb

@@ -0,0 +1,1004 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "view-in-github",
+        "colab_type": "text"
+      },
+      "source": [
+        "<a href=\"https://colab.research.google.com/github/vanderbilt-data-science/lo-achievement/blob/124-implement-baseline-functionality-for-oral-exam-module/UI_design_oral_exam_chatbot.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "PIbogPXyM0wr"
+      },
+      "source": [
+        "# Project IO Achievement - UI Design (Oral Exam)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "x_Vp8SiKM4p1"
+      },
+      "source": [
+        "## Problem Definition\n",
+        "\n",
+        "The v1 functionality for the Oral Exam module requires the following:\n",
+        "\n",
+        "1. Upload or generation of questions: either the user should upload a set of questions or we should allow the model to generate the questions. The user should pick or it should be inherent if there is no upload of questions. Note that we must also allow for context to be uploaded (vector store, vector store link, specific documents)\n",
+        "2. The model should prompt the user with a question and pause.\n",
+        "The user should respond by audio.\n",
+        "3. This should continue on until some final point where the exam is over.\n",
+        "\n",
+        "Then:\n",
+        "\n",
+        "1. We should use Whisper to do the transcription, and\n",
+        "2. Send the transcription, questions, and context for GPT4 for evaluation\n",
+        "Return the evaluation.\n",
+        "3. This will primarily be work on a user interface."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "o_60X8H3NEne"
+      },
+      "source": [
+        "## Libraries\n",
+        "\n",
+        "This section will install and import some important libraries such as Langchain, openai, Gradio, and so on"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 1,
+      "metadata": {
+        "id": "pxcqXgg2aAN7"
+      },
+      "outputs": [],
+      "source": [
+        "# install libraries here\n",
+        "# -q flag for \"quiet\" install\n",
+        "%%capture\n",
+        "!pip install -q langchain\n",
+        "!pip install -q openai\n",
+        "!pip install -q gradio\n",
+        "# !pip install -q datasets\n",
+        "!pip install -q torchaudio\n",
+        "!pip install -q git+https://github.com/openai/whisper.git\n",
+        "!pip install -q docx\n",
+        "!pip install -q PyPDF2\n",
+        "!pip install -q python-docx"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 2,
+      "metadata": {
+        "id": "pEjM1tLsMZBq"
+      },
+      "outputs": [],
+      "source": [
+        "# import libraries here\n",
+        "from langchain.llms import OpenAI\n",
+        "from langchain.prompts import PromptTemplate\n",
+        "from langchain.document_loaders import TextLoader\n",
+        "from langchain.indexes import VectorstoreIndexCreator\n",
+        "from langchain import ConversationChain, LLMChain, PromptTemplate\n",
+        "from langchain.chat_models import ChatOpenAI\n",
+        "from langchain.memory import ConversationBufferWindowMemory\n",
+        "from langchain.prompts import ChatPromptTemplate\n",
+        "from langchain.text_splitter import CharacterTextSplitter\n",
+        "from langchain.embeddings import OpenAIEmbeddings\n",
+        "import openai\n",
+        "import os\n",
+        "from getpass import getpass\n",
+        "# from IPython.display import display, Javascript, HTML\n",
+        "# from google.colab.output import eval_js\n",
+        "# from base64 import b64decode\n",
+        "# import ipywidgets as widgets\n",
+        "# from IPython.display import clear_output\n",
+        "import time\n",
+        "import requests\n",
+        "# from datasets import load_dataset\n",
+        "# from torchaudio.transforms import Resample\n",
+        "import whisper\n",
+        "import numpy as np\n",
+        "import torch\n",
+        "import librosa\n",
+        "# from datasets import load_dataset\n",
+        "#from jiwer import wer\n",
+        "import pandas as pd\n",
+        "import gradio as gr\n",
+        "from docx import Document\n",
+        "import PyPDF2\n",
+        "from pydub import AudioSegment\n",
+        "import tempfile"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "03KLZGI_a5W5"
+      },
+      "source": [
+        "## API Keys\n",
+        "\n",
+        "Use these cells to load the API keys required for this notebook. The below code cell uses the `getpass` library."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 3,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "5smcWj4DbFgy",
+        "outputId": "6bc91507-cd3c-4808-8976-811d7fc7cb29"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "··········\n"
+          ]
+        }
+      ],
+      "source": [
+        "openai_api_key = getpass()\n",
+        "os.environ[\"OPENAI_API_KEY\"] = openai_api_key\n",
+        "openai.api_key = openai_api_key"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "pMo9x8u4AEV1"
+      },
+      "source": [
+        "## Prompt Design"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 4,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "UgnCZRMhADvo",
+        "outputId": "462e62c7-a618-4549-e651-858514757235"
+      },
+      "outputs": [
+        {
+          "output_type": "execute_result",
+          "data": {
+            "text/plain": [
+              "ChatOpenAI(cache=None, verbose=False, callbacks=None, callback_manager=None, tags=None, metadata=None, client=<class 'openai.api_resources.chat_completion.ChatCompletion'>, model_name='gpt-4', temperature=0.0, model_kwargs={}, openai_api_key='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', openai_api_base='', openai_organization='', openai_proxy='', request_timeout=None, max_retries=6, streaming=False, n=1, max_tokens=None, tiktoken_model_name=None)"
+            ]
+          },
+          "metadata": {},
+          "execution_count": 4
+        }
+      ],
+      "source": [
+        "chat = ChatOpenAI(temperature=0.0, model_name='gpt-4')\n",
+        "chat"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "### Chatbot Prompts"
+      ],
+      "metadata": {
+        "id": "2tTNiyU-ZcDU"
+      }
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 15,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "r-VmK_7vHrmw",
+        "outputId": "8c314a8f-dad5-47b9-ddba-f6009a73b80d"
+      },
+      "outputs": [
+        {
+          "output_type": "execute_result",
+          "data": {
+            "text/plain": [
+              "['history', 'input', 'instruction', 'questions']"
+            ]
+          },
+          "metadata": {},
+          "execution_count": 15
+        }
+      ],
+      "source": [
+        "template_string3 = \"\"\"\n",
+        "Please ask me the following questions in sequence, and after I provide the answer, \\\n",
+        "please give me some feedback. Here is the instruction for feedback: {instruction}. If no instruction is provided, please provide feedback based on your judgement. \\\n",
+        "Just ask me the question, and please do not show any other text (no need for greetings for example) \\\n",
+        "Here are the questions that you can will me: {questions}. \\\n",
+        "Here are the chat history: {history}. \\\n",
+        "{input}\n",
+        "\n",
+        "Once all questions are answered, thank the user and give overall feedback for the question answering part.\n",
+        "\"\"\"\n",
+        "prompt_template3 = ChatPromptTemplate.from_template(template_string3)\n",
+        "prompt_template3.messages[0].prompt.input_variables"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 16,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/",
+          "height": 72
+        },
+        "id": "XaK7D5B4bMYv",
+        "outputId": "55f495b5-d8d0-4a80-b5c4-128dba34eebe"
+      },
+      "outputs": [
+        {
+          "output_type": "execute_result",
+          "data": {
+            "text/plain": [
+              "'\\nPlease ask me the following questions in sequence, and after I provide the answer, please give me some feedback. Here is the instruction for feedback: {instruction}. If no instruction is provided, please provide feedback based on your judgement. Just ask me the question, and please do not show any other text (no need for greetings for example) Here are the questions that you can will me: {questions}. Here are the chat history: {history}. {input}\\n\\nOnce all questions are answered, thank the user and give overall feedback for the question answering part.\\n'"
+            ],
+            "application/vnd.google.colaboratory.intrinsic+json": {
+              "type": "string"
+            }
+          },
+          "metadata": {},
+          "execution_count": 16
+        }
+      ],
+      "source": [
+        "prompt_template3.messages[0].prompt.template"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "l4o8R5eUE1n8"
+      },
+      "source": [
+        "### Functions"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 7,
+      "metadata": {
+        "id": "ABN0X9xQHeii"
+      },
+      "outputs": [],
+      "source": [
+        "def embed_key(openai_api_key):\n",
+        "  os.environ[\"OPENAI_API_KEY\"] = openai_api_key\n",
+        "\n",
+        "def transcribe(audio_file_path):\n",
+        "  try:\n",
+        "    with open(audio_file_path, \"rb\") as audio_file:\n",
+        "      # Call OpenAI's Whisper model for transcription\n",
+        "      transcript = openai.Audio.transcribe(\"whisper-1\", audio_file)\n",
+        "      transcribed_text = transcript[\"text\"]\n",
+        "      return transcribed_text\n",
+        "  except:\n",
+        "    return \"Your answer will be transcribed here\"\n",
+        "\n",
+        "def process_file(files):\n",
+        "  for file in files:\n",
+        "    try:\n",
+        "        extension = file.name.split('.')[-1].lower()\n",
+        "        if extension == 'docx':\n",
+        "            doc = Document(file.name)\n",
+        "            full_text = []\n",
+        "            for paragraph in doc.paragraphs:\n",
+        "                full_text.append(paragraph.text)\n",
+        "            return '\\n'.join(full_text)\n",
+        "\n",
+        "        elif extension == 'pdf':\n",
+        "            pdf_file = open(file.name, 'rb')\n",
+        "            reader = PyPDF2.PdfReader(pdf_file)\n",
+        "            num_pages = len(reader.pages)\n",
+        "            full_text = []\n",
+        "            for page in range(num_pages):\n",
+        "                page_obj = reader.pages[page]\n",
+        "                full_text.append(page_obj.extract_text())\n",
+        "            pdf_file.close()\n",
+        "            return '\\n'.join(full_text)\n",
+        "\n",
+        "        elif extension == 'txt':\n",
+        "            with open(file.name, 'r') as txt_file:\n",
+        "                full_text = txt_file.read()\n",
+        "            return full_text\n",
+        "\n",
+        "        else:\n",
+        "            return \"Unsupported file type\"\n",
+        "    except FileNotFoundError:\n",
+        "        return \"File not found\"\n",
+        "    except PermissionError:\n",
+        "        return \"Permission denied\"\n",
+        "\n",
+        "def generate_questions(text, prompt):\n",
+        "    test_input1 = question_template.format_messages(\n",
+        "                      context = text,\n",
+        "                      pre_prompt = prompt)\n",
+        "\n",
+        "    response = chat(test_input1)\n",
+        "    return response.content\n",
+        "\n",
+        "\n",
+        "def ai_evaluate(context, audio_transcript, QA, instructions):\n",
+        "    test_input1 = evaluate_template.format_messages(\n",
+        "                      context = context,\n",
+        "                      transcript = audio_transcript,\n",
+        "                      QA = QA,\n",
+        "                      instructions = instructions)\n",
+        "\n",
+        "    response = chat(test_input1)\n",
+        "    return response.content\n",
+        "\n",
+        "def upload_file(files):\n",
+        "    file_paths = [file.name for file in files]\n",
+        "    return file_paths\n",
+        "\n",
+        "def use_these_questions(input):\n",
+        "    return input\n",
+        "\n",
+        "################################\n",
+        "\n",
+        "def add_text(history, text, prompt = template_string3):\n",
+        "    new_history = [(prompt, None)] + history + [(text, None)]\n",
+        "    return new_history, gr.update(value=\"\", interactive=False)\n",
+        "\n",
+        "# def add_file(history, file):\n",
+        "#     history = history + [((file.name,), None)]\n",
+        "#     return history\n",
+        "\n",
+        "\n",
+        "def bot_initialize(input, instruction_feedback, questions_used, history):\n",
+        "\n",
+        "    template_string3 = \"\"\"\n",
+        "    Please ask me the following questions in sequence, and after I provide the answer, \\\n",
+        "    please give me some feedback. Here is the instruction for feedback: {instruction}. If no instruction is provided, please provide feedback based on your judgement. \\\n",
+        "    Here are the questions that you can ask me: {questions}. \\\n",
+        "    Here are the chat history: {history}. \\\n",
+        "    {input} \\\n",
+        "\n",
+        "    *** Remember, just ask me the question, give feedbacks, and ask the next questions. Do not forget to ask the next question after feedbacks. \\\n",
+        "    \"\"\"\n",
+        "    prompt_template3 = ChatPromptTemplate.from_template(template_string3)\n",
+        "\n",
+        "    test_input1 = prompt_template3.format_messages(\n",
+        "                      instruction = instruction_feedback,\n",
+        "                      history = history,\n",
+        "                      questions = questions_used,\n",
+        "                      input = input)\n",
+        "\n",
+        "    response = chat(test_input1)\n",
+        "    return response.content\n",
+        "\n",
+        "# def initialize(instruction_feedback, questions_used, chat_history, ready):\n",
+        "#     test_input1 = prompt_template3.format_messages(\n",
+        "#                       instruction = instruction_feedback,\n",
+        "#                       chat_history = chat_history,\n",
+        "#                       questions = questions_used,\n",
+        "#                       ready = ready)\n",
+        "#     response = chat(test_input1)\n",
+        "#     return response.content\n",
+        "\n",
+        "# def bot(history):\n",
+        "#     response = \"**That's cool!**\"\n",
+        "#     history[-1][1] = \"\"\n",
+        "#     for character in response:\n",
+        "#         history[-1][1] += character\n",
+        "#         time.sleep(0.05)\n",
+        "#         yield history\n",
+        "\n",
+        "def message_and_history(input, instruction_feedback, questions_used, history):\n",
+        "    history = history or []\n",
+        "    s = list(sum(history, ()))\n",
+        "    s.append(input)\n",
+        "    inp = ' '.join(s)\n",
+        "    output = bot_initialize(inp, instruction_feedback, questions_used, history)\n",
+        "    history.append((input, output))\n",
+        "    return history, history\n",
+        "\n",
+        "def prompt_select(selection, number, length):\n",
+        "  if selection == \"Random\":\n",
+        "    prompt = f\"Please design a {number} question quiz based on the context provided and the inputted learning objectives (if applicable).\"\n",
+        "  elif selection == \"Fill in the Blank\":\n",
+        "    prompt = f\"Create a {number} question fill in the blank quiz refrencing the context provided. The quiz should reflect the learning objectives (if inputted). The 'blank' part of the question should appear as '________'. The answers should reflect what word(s) should go in the blank an accurate statement. An example is the follow: 'The author of the article is ______.' The question should be a statement.\"\n",
+        "  elif selection == \"Short Answer\":\n",
+        "    prompt = f\"Please design a {number} question quiz about which reflects the learning objectives (if inputted).  The questions should be short answer. Expect the correct answers to be {length} sentences long.\"\n",
+        "  else:\n",
+        "    prompt = f\"Please design a {number} question {selection.lower()} quiz based on the context provided and the inputted learning objectives (if applicable).\"\n",
+        "  return prompt\n",
+        "\n",
+        "# def prompt_select(selection, number, length):\n",
+        "#   if selection == \"Random\":\n",
+        "#     prompt = f\"Please design a {number} question quiz based on the context provided and the inputted learning objectives (if applicable). The types of questions should be randomized (including multiple choice, short answer, true/false, short answer, etc.).  Provide one question at a time, and wait for my response before providing me with feedback. Again, while the quiz may ask for multiple questions, you should only provide 1 question in you initial response. Do not include the answer in your response. If I get an answer wrong, provide me with an explanation of why it was incorrect, and then give me additional chances to respond until I get the correct choice. Explain why the correct choice is right.\"\n",
+        "#   elif selection == \"Fill in the Blank\":\n",
+        "#     prompt = f\"Create a {number} question fill in the blank quiz refrencing the context provided. The quiz should reflect the learning objectives (if inputted). The 'blank' part of the question should appear as '________'. The answers should reflect what word(s) should go in the blank an accurate statement. An example is the follow: 'The author of the article is ______.' The question should be a statement. Provide one question at a time, and wait for my response before providing me with feedback. Again, while the quiz may ask for multiple questions, you should only provide ONE question in you initial response. Do not include the answer in your response. If I get an answer wrong, provide me with an explanation of why it was incorrect,and then give me additional chances to respond until I get the correct choice. Explain why the correct choice is right.\"\n",
+        "#   elif selection == \"Short Answer\":\n",
+        "#     prompt = f\"Please design a {number} question quiz about which reflects the learning objectives (if inputted).  The questions should be short answer. Expect the correct answers to be {length} sentences long. Provide one question at a time, and wait for my response before providing me with feedback. Again, while the quiz may ask for multiple questions, you should only provide ONE question in you initial response. Do not include the answer in your response. If I get an answer wrong, provide me with an explanation of why it was incorrect, and then give me additional chances to respond until I get the correct choice. Explain why the correct answer is right.\"\n",
+        "#   else:\n",
+        "#     prompt = f\"Please design a {number} question {selection.lower()} quiz based on the context provided and the inputted learning objectives (if applicable). Provide one question at a time, and wait for my response before providing me with feedback. Again, while the quiz may ask for multiple questions, you should only provide 1 question in you initial response. Do not include the answer in your response. If I get an answer wrong, provide me with an explanation of why it was incorrect, and then give me additional chances to respond until I get the correct choice. Explain why the correct choice is right.\"\n",
+        "#   return prompt"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "8PzIpcfg4-X0"
+      },
+      "source": [
+        "## Integrate Prompts from LO project"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "E5vWxcm25EAC"
+      },
+      "source": [
+        "### Creating a Chain for Short Answer Generation"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "y95FExV-5IqI"
+      },
+      "source": [
+        "In this example, the context would include the poem \"The Road Not Taken\" by Robert Frost"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 8,
+      "metadata": {
+        "id": "j_qEXDWQ5RSW"
+      },
+      "outputs": [],
+      "source": [
+        "# This is what I used to test the function 'generate_questions_v2'\n",
+        "template_string = \"\"\"\n",
+        "You are a world-class tutor helping students to perform better on oral and written exams though interactive experiences.\"\n",
+        "\n",
+        "The following text should be used as the basis for the instructions which follow: {context} \\\n",
+        "\n",
+        "The following is the guideline for generating the questiion: {pre_prompt} \\\n",
+        "\n",
+        "The output should be formatted as following:\n",
+        "\n",
+        "Question 1: ...\n",
+        "Question 2: ...\n",
+        "Question 3: ...\n",
+        "...\n",
+        "\"\"\""
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 9,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "yWOM1XdC5UhQ",
+        "outputId": "69d781f7-fb0c-4dde-9085-ddb9908b82af"
+      },
+      "outputs": [
+        {
+          "output_type": "execute_result",
+          "data": {
+            "text/plain": [
+              "['context', 'pre_prompt']"
+            ]
+          },
+          "metadata": {},
+          "execution_count": 9
+        }
+      ],
+      "source": [
+        "question_template = ChatPromptTemplate.from_template(template_string)\n",
+        "question_template.messages[0].prompt.input_variables"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 10,
+      "metadata": {
+        "id": "4Mc1ZC3jaydQ"
+      },
+      "outputs": [],
+      "source": [
+        "# @title\n",
+        "con = \"\"\" Two roads diverged in a yellow wood,\n",
+        "And sorry I could not travel both\n",
+        "And be one traveler, long I stood\n",
+        "And looked down one as far as I could\n",
+        "To where it bent in the undergrowth;\n",
+        "Then took the other, as just as fair,\n",
+        "And having perhaps the better claim,\n",
+        "Because it was grassy and wanted wear;\n",
+        "Though as for that the passing there\n",
+        "Had worn them really about the same,\n",
+        "And both that morning equally lay\n",
+        "In leaves no step had trodden black.\n",
+        "Oh, I kept the first for another day!\n",
+        "Yet knowing how way leads on to way,\n",
+        "I doubted if I should ever come back.\n",
+        "I shall be telling this with a sigh\n",
+        "Somewhere ages and ages hence:\n",
+        "Two roads diverged in a wood, and I—\n",
+        "I took the one less traveled by,\n",
+        "And that has made all the difference.\n",
+        "—-Robert Frost—-\n",
+        "Education Place: http://www.eduplace.com \"\"\"\n",
+        "\n",
+        "pre = \"Please design a 3 question quiz about which reflects the learning objectives (if inputted).  The questions should be short answer. Expect the correct answers to be  sentences long. Provide one question at a time, and wait for my response before providing me with feedback. Again, while the quiz may ask for multiple questions, you should only provide ONE question in you initial response. Do not include the answer in your response. If I get an answer wrong, provide me with an explanation of why it was incorrect, and then give me additional chances to respond until I get the correct choice. Explain why the correct answer is right.\"\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 11,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/",
+          "height": 36
+        },
+        "id": "KXssPFyEbG3f",
+        "outputId": "8a5389f7-3ec6-4332-d9ae-f45ff5781caf"
+      },
+      "outputs": [
+        {
+          "output_type": "execute_result",
+          "data": {
+            "text/plain": [
+              "'Question 1: What is the main theme of Robert Frost\\'s poem \"The Road Not Taken\"?'"
+            ],
+            "application/vnd.google.colaboratory.intrinsic+json": {
+              "type": "string"
+            }
+          },
+          "metadata": {},
+          "execution_count": 11
+        }
+      ],
+      "source": [
+        "generate_questions(con,pre)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "DMTybR3PVuoC"
+      },
+      "source": [
+        "### Creating a Chain for AI Evaluation"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 12,
+      "metadata": {
+        "id": "Wc-3XAFQVxO_"
+      },
+      "outputs": [],
+      "source": [
+        "template_evaluation = \"\"\"\n",
+        "Given\n",
+        "1. The follwing context of the oral exam/presentation: {context} \\\n",
+        "\n",
+        "2. The answer from the student: {transcript} \\\n",
+        "\n",
+        "3. The Questions asked to the student and student answers {QA} \\\n",
+        "\n",
+        "Please evaluate the students performance based on {instructions} \\\n",
+        "\n",
+        "If no instruction is provided, you can evaluate based on your judgement of the students performance.\n",
+        "\n",
+        "\"\"\""
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 13,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "FZXeYNSVVy5g",
+        "outputId": "94a1a4be-03a6-4c20-97b8-4333910991fa"
+      },
+      "outputs": [
+        {
+          "output_type": "execute_result",
+          "data": {
+            "text/plain": [
+              "['QA', 'context', 'instructions', 'transcript']"
+            ]
+          },
+          "metadata": {},
+          "execution_count": 13
+        }
+      ],
+      "source": [
+        "# @title\n",
+        "evaluate_template = ChatPromptTemplate.from_template(template_evaluation)\n",
+        "evaluate_template.messages[0].prompt.input_variables"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "a3WUL_hFyMkr"
+      },
+      "source": [
+        "### Test process_file"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/",
+          "height": 157
+        },
+        "id": "LJX1AKTMyVm8",
+        "outputId": "24f8de7a-f456-47bc-b3b5-0284bbd7076f"
+      },
+      "outputs": [
+        {
+          "data": {
+            "application/vnd.google.colaboratory.intrinsic+json": {
+              "type": "string"
+            },
+            "text/plain": [
+              "\"\\ufeffHello,\\n\\n\\nWe are so excited for this semester’s partnership with Data Science Institute and Next Steps at Vanderbilt. Jonathan Wade will be interning Mondays and Wednesdays 2-5 pm and Fridays 12-4 pm starting Monday, January 31st. Jessica will be job coaching on Mondays and Wednesdays from 2-5 pm. It also used to be on Fridays from 2-4 pm but not anymore.\\n\\n\\nBelow is important information and reminders:\\n\\n\\nLocation: 1400 18th Ave S Building, Suite 2000, Nashville, TN 37212\\n\\n\\nDress: Business casual attire (This includes items such as dress pants, khakis, polos and dress shirts)\\n\\n\\nImportant Dates:\\n\\n\\nVanderbilt University is off for Spring Break March 5th - March 13th. No internships will take place these days.\\n\\n\\nAll internships end by Thursday, April 28th.\\n\\n\\nImportant COVID-19 Information: Attached is a document outlining the COVID-19 guidelines all Vanderbilt University students must follow, including Next Steps interns and job coaches while at internship sites. Please note these may change given the evolving nature of the pandemic and any changes will be communicated to internship sites, interns, and job coaches as needed.\\n\\n\\nCareer Development Resource Guide: I am also attaching the Next Steps Career Development Resource guide that outlines student expectations and provides helpful information and resources for site supervisors.\\n\\n\\nInternship Coordinator: Lynda Tricia is the coordinator for this internship and is the main point of contact for any questions or concerns that arise.\\n\\n\\nFinally, below you will find everyone's contact information for your convenience.\\n\\n\\nContacts:\\n\\n\\nIntern: Jonathan Wade, jonathan.j.wade@vanderbilt.edu, 613-472-3867\\n\\n\\nSupervisor: Ruben Miller, ruben.k.miller@vanderbilt.edu, 216-574-3176\\n\\n\\nJob Coach: Jessica Cho, jessica.cho@vanderbilt.edu, 615-999-1134\\n\\n\\nInternship Coordinator: Lynda Tricia, lynda.z.tricia@vanderbilt.edu, 606-415-9999\\n\\n\\nPlease let us know if there are any questions. Thank you!\\n\\n\\n\\n\\nNext Steps at Vanderbilt - Safety Guidelines for Internships\\nMore information on Vanderbilt’s Health and Safety Protocols can be found here: https://www.vanderbilt.edu/coronavirus/community/undergraduate-students/\\nMasks: All Next Steps interns and job coaches must wear masks indoors at internships even if the jobsite does not require masks.\\nInterns and job coaches should have a well-fitted mask that completely covers your nose and mouth, preferably a KN95, KF94 or FFP2 version.\\nLunch Breaks: If an intern or job coach needs a lunch break, they can remove their mask just when they eat or drink. They must be physically distanced from other co-workers when eating. \\nSymptom Monitoring: All interns and job coaches must be free of ANY symptoms related to COVID-19 to go to the internship. If an intern or job coach has symptoms, they should stay home, notify the Next Steps staff and internship supervisor, and get tested at Vanderbilt Student Health or with their medical provider.\\nAccording to the CDC, symptoms may appear 2 to 14 days after exposure to the virus. These include:\\n* Fever or chills\\n* Cough\\n* Shortness of breath or difficulty breathing\\n* Fatigue\\n* Muscle or body aches\\n* Headache\\n* New loss of taste or smell\\n* Sore throat\\n* Congestion or runny nose\\n* Nausea or vomiting\\n* Diarrhea\\n\\n\\nIf an intern or job coach tests positive: If an intern or job coach receives a COVID-19 positive test result, regardless of vaccination status, they should complete the following webform. The webform goes directly to the Vanderbilt Command Center.\\nThe intern or job coach will receive direct communication from the Command Center about their isolation (if they tested positive) or quarantine period (if considered a close contact) and will be instructed to contact Student Health or Occupational Health if they develop symptoms.\\nClose Contact/Quarantine: Interns or job coaches who are a close contact of someone who tests positive should complete the following webform. The webform goes directly to the Command Center.\\n* Close contacts who are unvaccinated will quarantine for 10 days based on CDC guidance.\\no Additional requirements are in place for days 10 to 14 following exposure including:\\n* For days 10 to 14 after last exposure, unvaccinated Vanderbilt community members identified as close contacts must not unmask at any time in public.\\n* Individuals should eat alone or complete any activities alone that require removing a mask in a private space during those four days between day 10-14.\\n* Close contacts who are vaccinated and asymptomatic will not have to quarantine but are recommended to monitor their symptoms and to get a COVID-19 test 5-7 days after last exposure. If asymptomatic, testing can be done at the VU testing center. If individuals develop symptoms, they should test at Student Health, Occupational Health or VUMC or other testing location in the community.\\n* Close contacts who are vaccinated and symptomatic may have to quarantine based on severity of symptoms and specific living situations. This determination will be made by their medical provider in consultation with the Command Center.\\nJob Coaching Supports: If a job coach tests positive and is unable to provide on-site supports, Next Steps staff will follow the procedures outlined below.\\n1. Identify another job coach or Next Steps staff member who can provide job coaching on-site during some of the student’s internship hours.\\n2. Identify another job coach or staff member who can check-in virtually with the student and supervisor during their shift. \\n3. Or, work with the student and supervisor to ensure natural supports are in place if the student must work without support of a job coach during that time period.\""
+            ]
+          },
+          "execution_count": 69,
+          "metadata": {},
+          "output_type": "execute_result"
+        }
+      ],
+      "source": [
+        "# Might need some way to make pdf file to load more readable\n",
+        "# process_file('/content/instrutor_note.docx')\n",
+        "# process_file('/content/Big Data & Economics.pdf')\n",
+        "# process_file('/content/Big Data & Economics.pdf')\n",
+        "process_file('/content/Anonymized Job Coach QA Test Doc (1).txt')"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "M6IzVTjz5cex"
+      },
+      "source": [
+        "## UI Design\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "u2SY4Akt_t8h"
+      },
+      "source": [
+        "### Chatbot V2"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 17,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/",
+          "height": 853
+        },
+        "id": "6ENnsKlD_uOC",
+        "outputId": "7b694aa8-e314-4502-a590-3245ebc3e1e0"
+      },
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "Colab notebook detected. This cell will run indefinitely so that you can see errors and logs. To turn off, set debug=False in launch().\n",
+            "Note: opening Chrome Inspector may crash demo inside Colab notebooks.\n",
+            "\n",
+            "To create a public link, set `share=True` in `launch()`.\n"
+          ]
+        },
+        {
+          "output_type": "display_data",
+          "data": {
+            "text/plain": [
+              "<IPython.core.display.Javascript object>"
+            ],
+            "application/javascript": [
+              "(async (port, path, width, height, cache, element) => {\n",
+              "                        if (!google.colab.kernel.accessAllowed && !cache) {\n",
+              "                            return;\n",
+              "                        }\n",
+              "                        element.appendChild(document.createTextNode(''));\n",
+              "                        const url = await google.colab.kernel.proxyPort(port, {cache});\n",
+              "\n",
+              "                        const external_link = document.createElement('div');\n",
+              "                        external_link.innerHTML = `\n",
+              "                            <div style=\"font-family: monospace; margin-bottom: 0.5rem\">\n",
+              "                                Running on <a href=${new URL(path, url).toString()} target=\"_blank\">\n",
+              "                                    https://localhost:${port}${path}\n",
+              "                                </a>\n",
+              "                            </div>\n",
+              "                        `;\n",
+              "                        element.appendChild(external_link);\n",
+              "\n",
+              "                        const iframe = document.createElement('iframe');\n",
+              "                        iframe.src = new URL(path, url).toString();\n",
+              "                        iframe.height = height;\n",
+              "                        iframe.allow = \"autoplay; camera; microphone; clipboard-read; clipboard-write;\"\n",
+              "                        iframe.width = width;\n",
+              "                        iframe.style.border = 0;\n",
+              "                        element.appendChild(iframe);\n",
+              "                    })(7860, \"/\", \"100%\", 500, false, window.element)"
+            ]
+          },
+          "metadata": {}
+        },
+        {
+          "output_type": "stream",
+          "name": "stderr",
+          "text": [
+            "/usr/local/lib/python3.10/dist-packages/gradio/processing_utils.py:188: UserWarning: Trying to convert audio automatically from int32 to 16-bit int format.\n",
+            "  warnings.warn(warning.format(data.dtype))\n",
+            "/usr/local/lib/python3.10/dist-packages/gradio/processing_utils.py:188: UserWarning: Trying to convert audio automatically from int32 to 16-bit int format.\n",
+            "  warnings.warn(warning.format(data.dtype))\n",
+            "/usr/local/lib/python3.10/dist-packages/gradio/processing_utils.py:188: UserWarning: Trying to convert audio automatically from int32 to 16-bit int format.\n",
+            "  warnings.warn(warning.format(data.dtype))\n",
+            "/usr/local/lib/python3.10/dist-packages/gradio/processing_utils.py:188: UserWarning: Trying to convert audio automatically from int32 to 16-bit int format.\n",
+            "  warnings.warn(warning.format(data.dtype))\n",
+            "/usr/local/lib/python3.10/dist-packages/gradio/processing_utils.py:188: UserWarning: Trying to convert audio automatically from int32 to 16-bit int format.\n",
+            "  warnings.warn(warning.format(data.dtype))\n"
+          ]
+        },
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "Keyboard interruption in main thread... closing server.\n"
+          ]
+        },
+        {
+          "output_type": "execute_result",
+          "data": {
+            "text/plain": [
+              "'\\nWhat Are the Different Types of Machine Learning?\\nHow Do You Handle Missing or Corrupted Data in a Dataset?\\nHow Can You Choose a Classifier Based on a Training Set Data Size?\\nExplain the Confusion Matrix with Respect to Machine Learning Algorithms.\\nWhat Are the Differences Between Machine Learning and Deep Learning\\n'"
+            ],
+            "application/vnd.google.colaboratory.intrinsic+json": {
+              "type": "string"
+            }
+          },
+          "metadata": {},
+          "execution_count": 17
+        }
+      ],
+      "source": [
+        "with gr.Blocks(theme=gr.themes.Monochrome()) as demo:\n",
+        "  gr.Markdown(\"# Oral Exam App\")\n",
+        "  gr.Markdown(\"## OpenAI API key\")\n",
+        "  with gr.Box():\n",
+        "    gr.HTML(\"\"\"Embed your OpenAI API key below; if you haven't created one already, visit\n",
+        "      platform.openai.com/account/api-keys\n",
+        "    to sign up for an account and get your personal API key\"\"\",\n",
+        "            elem_classes=\"textbox_label\")\n",
+        "    input = gr.Textbox(show_label=False, type=\"password\", container=False,\n",
+        "                      placeholder=\"●●●●●●●●●●●●●●●●●\")\n",
+        "    input.change(fn=embed_key, inputs=input, outputs=None)\n",
+        "\n",
+        "  with gr.Blocks():\n",
+        "  #########################\n",
+        "  #########Context#########\n",
+        "  #########################\n",
+        "      with gr.Accordion(\"Context section\"):\n",
+        "      ### Should also allow vector stores\n",
+        "          gr.Markdown(\"## Please upload the context document(s) for Oral exam\")\n",
+        "          context_input = gr.File(label=\"Click to upload context file\",\n",
+        "                                  file_count=\"multiple\",\n",
+        "                                  file_types=[\".txt\", \".docx\", \".pdf\"])\n",
+        "          outputs_context=gr.Textbox(label=\"Context\")\n",
+        "          context_input.change(fn=process_file, inputs=context_input, outputs=outputs_context)\n",
+        "          # upload_button = gr.Button(value=\"Show context\")\n",
+        "          # upload_button.click(process_file, context_input, outputs_context)\n",
+        "\n",
+        "      with gr.Blocks():\n",
+        "        gr.Markdown(\"\"\"\n",
+        "        ## Generate a Premade Prompt\n",
+        "        Select your type and number of desired questions. Click \"Generate Prompt\" to get your premade prompt,\n",
+        "        and then \"Insert Prompt into Chat\" to copy the text into the chat interface below. \\\n",
+        "        You can also copy the prompt using the icon in the upper right corner and paste directly into the input box when interacting with the model.\n",
+        "        \"\"\")\n",
+        "        with gr.Row():\n",
+        "          with gr.Column():\n",
+        "            question_type = gr.Dropdown([\"Multiple Choice\", \"True or False\", \"Short Answer\", \"Fill in the Blank\", \"Random\"], label=\"Question Type\")\n",
+        "            number_of_questions = gr.Textbox(label=\"Enter desired number of questions\")\n",
+        "            sa_desired_length = gr.Dropdown([\"1-2\", \"3-4\", \"5-6\", \"6 or more\"], label = \"For short answer questions only, choose the desired sentence length for answers. The default value is 1-2 sentences.\")\n",
+        "          with gr.Column():\n",
+        "            prompt_button = gr.Button(\"Generate Prompt\")\n",
+        "            premade_prompt_output = gr.Textbox(label=\"Generated prompt (save or copy)\", show_copy_button=True)\n",
+        "            prompt_button.click(prompt_select,\n",
+        "                            inputs=[question_type, number_of_questions, sa_desired_length],\n",
+        "                            outputs=premade_prompt_output)\n",
+        "\n",
+        "  #########################\n",
+        "  #######Main Audio########\n",
+        "  #########################\n",
+        "      with gr.Accordion(\"Main audio section\"):\n",
+        "        gr.Markdown(\"## Upload your audio file or start recording\")\n",
+        "\n",
+        "        with gr.Column():\n",
+        "          with gr.Row():\n",
+        "              file_input = gr.Audio(label=\"Upload Audio\", source=\"upload\", type=\"filepath\")\n",
+        "              record_inputs = gr.Audio(label=\"Record Audio\", source=\"microphone\", type=\"filepath\")\n",
+        "\n",
+        "          gr.Markdown(\"## Transcribe the audio uploaded or recorded\")\n",
+        "          outputs_transcribe=gr.Textbox(label=\"Transcription\")\n",
+        "\n",
+        "          file_input.change(fn=transcribe, inputs=file_input, outputs=outputs_transcribe)\n",
+        "          record_inputs.change(fn=transcribe, inputs=record_inputs, outputs=outputs_transcribe)\n",
+        "\n",
+        "  # #########################\n",
+        "  # ###Question Generation###\n",
+        "  # #########################\n",
+        "  #     with gr.Accordion(\"Question section\"):\n",
+        "  #         gr.Markdown(\"## Questions\")\n",
+        "  #         with gr.Column():\n",
+        "  #           outputs_qa=gr.Textbox(label=\"Generate questions or Use your own questions\")\n",
+        "  #           btn3 = gr.Button(value=\"Generate questions\")\n",
+        "  #           btn3.click(generate_questions, inputs=context_input, outputs=outputs_qa)\n",
+        "\n",
+        "\n",
+        "  ########################\n",
+        "  ##Question Generation###\n",
+        "  ########################\n",
+        "      with gr.Accordion(\"Question section\"):\n",
+        "          gr.Markdown(\"## Questions\")\n",
+        "          with gr.Row():\n",
+        "            with gr.Column():\n",
+        "              outputs_qa=gr.Textbox(label=\"Generate questions or Use your own questions\")\n",
+        "              btn1 = gr.Button(value=\"Generate questions\")\n",
+        "              btn1.click(generate_questions, inputs=[outputs_context, premade_prompt_output], outputs=outputs_qa)\n",
+        "\n",
+        "            # with gr.Column():\n",
+        "            #   submit_question=gr.Textbox(label=\"Use existing questions\")\n",
+        "            #   btn4 = gr.Button(value=\"Use these questions\")\n",
+        "              # btn4.click(use_this_question, inputs=outputs_transcribe, outputs=None)\n",
+        "\n",
+        "\n",
+        "  #########################\n",
+        "  #######Instruction#######\n",
+        "  #########################\n",
+        "          instruction_qa_input = gr.File(label=\"Click to upload instruction file\",\n",
+        "                                  file_count=\"multiple\",\n",
+        "                                  file_types=[\".txt\", \".docx\", \".pdf\"])\n",
+        "          instruction_qa=gr.Textbox(label=\"Or please enter the instruction for question/answering section\")\n",
+        "          instruction_qa.change(fn=process_file, inputs=context_input, outputs=outputs_context)\n",
+        "\n",
+        "\n",
+        "  #########################\n",
+        "  #########Audio QA########\n",
+        "  #########################\n",
+        "      with gr.Accordion(\"Audio QA section\"):\n",
+        "          gr.Markdown(\"## Question answering\")\n",
+        "          gr.Markdown(\"### When you are ready to answer questions, press the 'I am ready' button\")\n",
+        "          ##### This may be iterative\n",
+        "          chatbot = gr.Chatbot([],\n",
+        "                                  elem_id=\"chatbot\",\n",
+        "                                  height=300)\n",
+        "          state = gr.State()\n",
+        "          message = gr.Textbox(show_label=False,\n",
+        "                              placeholder=\"Your answer will be transcribed here\",\n",
+        "                              container=False)\n",
+        "          ready_button = gr.Button(value=\"I am ready\")\n",
+        "          ready_button.click(message_and_history, inputs=[message, instruction_qa, outputs_qa, state], outputs=[chatbot, state])\n",
+        "\n",
+        "          hidden = gr.Textbox(visible = False)\n",
+        "          btn_record = gr.Audio(label=\"Record Audio\", source=\"microphone\", type=\"filepath\")\n",
+        "          btn_record.change(fn=transcribe, inputs=btn_record, outputs=message)\n",
+        "          btn_record.clear(use_these_questions, inputs = hidden, outputs = message)\n",
+        "\n",
+        "          submit = gr.Button(\"Submit\")\n",
+        "          submit.click(message_and_history,\n",
+        "                      inputs=[message, instruction_qa, outputs_qa, state],\n",
+        "                      outputs=[chatbot, state])\n",
+        "\n",
+        "          message_records = gr.Textbox(show_label=False,\n",
+        "                              container=False)\n",
+        "          show_records = gr.Button(\"Show QA history\")\n",
+        "          show_records.click(use_these_questions,\n",
+        "                      inputs=state,\n",
+        "                      outputs=message_records)\n",
+        "\n",
+        "  #########################\n",
+        "  #######Evaluation########\n",
+        "  #########################\n",
+        "      with gr.Accordion(\"Evaluation section\"):\n",
+        "          gr.Markdown(\"## Evaluation\")\n",
+        "          with gr.Tab(\"General evalution\"):\n",
+        "            evalution=gr.Textbox(label=\"AI Evaluation\")\n",
+        "            btn5 = gr.Button(value=\"Evaluate\")\n",
+        "            btn5.click(ai_evaluate, inputs=[outputs_context, outputs_transcribe, message_records, instruction_qa], outputs=evalution)\n",
+        "          with gr.Tab(\"Quantitative evalution\"):\n",
+        "            table_output = gr.Dataframe(label = \"Some kind of evaluation metrics?\")\n",
+        "            btn6 = gr.Button(value=\"Evaluate\")\n",
+        "            # btn6.click(ai_evaluate, inputs=[outputs_context, message_records, outputs_qa], outputs=table_output)\n",
+        "\n",
+        "  # demo.launch()\n",
+        "  # demo.launch(share=True)\n",
+        "  demo.launch(debug=True)\n",
+        "\n",
+        "'''\n",
+        "What Are the Different Types of Machine Learning?\n",
+        "How Do You Handle Missing or Corrupted Data in a Dataset?\n",
+        "How Can You Choose a Classifier Based on a Training Set Data Size?\n",
+        "Explain the Confusion Matrix with Respect to Machine Learning Algorithms.\n",
+        "What Are the Differences Between Machine Learning and Deep Learning\n",
+        "'''"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "g2EVIogW69Fd"
+      },
+      "source": [
+        "## What's left\n",
+        "- vector store (link) upload\n",
+        "- how to not show the warning when transcribing\n",
+        "- better prompt for evaluation\n",
+        "- try ChatInterface of Gradio"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [],
+      "metadata": {
+        "id": "-YwOAtNANrx_"
+      },
+      "execution_count": null,
+      "outputs": []
+    }
+  ],
+  "metadata": {
+    "colab": {
+      "provenance": [],
+      "include_colab_link": true
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}

exploring_other_media_document_sources.ipynb

@@ -0,0 +1,459 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "colab_type": "text",
+        "id": "view-in-github"
+      },
+      "source": [
+        "<a href=\"https://colab.research.google.com/github/vanderbilt-data-science/lo-achievement/blob/main/exploring_other_media_document_sources.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "x_Vp8SiKM4p1"
+      },
+      "source": [
+        "# Exploring Alternative Media Document Sources\n",
+        "Test how one could get YouTube videos or websites as sources for documents in a vector store.\n",
+        "\n",
+        "- YouTube: https://python.langchain.com/docs/modules/data_connection/document_loaders/integrations/youtube_audio\n",
+        "- Websites:\n",
+        "  - https://js.langchain.com/docs/modules/indexes/document_loaders/examples/web_loaders/\n",
+        "  - https://python.langchain.com/docs/modules/data_connection/document_loaders/integrations/web_base\n",
+        "  - Extracting relevant information from website: https://www.oncrawl.com/technical-seo/extract-relevant-text-content-from-html-page/\n",
+        "\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "o_60X8H3NEne"
+      },
+      "source": [
+        "## Libraries"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "pxcqXgg2aAN7",
+        "outputId": "0bb1c0aa-99f7-4d8d-a66f-992ea54eb5ff"
+      },
+      "outputs": [],
+      "source": [
+        "# install libraries here\n",
+        "# -q flag for \"quiet\" install\n",
+        "!pip install -q langchain\n",
+        "!pip install -q openai\n",
+        "!pip install -q unstructured\n",
+        "!pip install -q tiktoken\n",
+        "!pip install typing_extensions==4.5.0"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/",
+          "height": 784
+        },
+        "id": "mwpl3jYJoGo7",
+        "outputId": "cbae7a1a-b4a4-4d32-f837-dcafb3d01746"
+      },
+      "outputs": [],
+      "source": [
+        "%pip install -q trafilatura\n",
+        "%pip install -q justext"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "NU-7ynWHvwfM",
+        "outputId": "cdd9a4db-1bd0-471c-d31e-e0c16aff0c98"
+      },
+      "outputs": [],
+      "source": [
+        "%pip install yt_dlp\n",
+        "%pip install pydub"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 2,
+      "metadata": {
+        "id": "pEjM1tLsMZBq"
+      },
+      "outputs": [],
+      "source": [
+        "# import libraries here\n",
+        "import os\n",
+        "import time\n",
+        "import pprint\n",
+        "from getpass import getpass\n",
+        "\n",
+        "from langchain.docstore.document import Document\n",
+        "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
+        "from langchain.embeddings import OpenAIEmbeddings\n",
+        "\n",
+        "from langchain.document_loaders.unstructured import UnstructuredFileLoader"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 5,
+      "metadata": {
+        "id": "0U6N_9xFsOcw"
+      },
+      "outputs": [],
+      "source": [
+        "from langchain.document_loaders.generic import GenericLoader\n",
+        "from langchain.document_loaders.parsers import OpenAIWhisperParser\n",
+        "from langchain.document_loaders.blob_loaders.youtube_audio import YoutubeAudioLoader"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 40,
+      "metadata": {
+        "id": "JRw367IwryWd"
+      },
+      "outputs": [],
+      "source": [
+        "from langchain.document_loaders import WebBaseLoader\n",
+        "import trafilatura\n",
+        "import requests\n",
+        "import justext"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "n0BTyPI_srMg"
+      },
+      "source": []
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 3,
+      "metadata": {
+        "id": "NOX639OA2pOh"
+      },
+      "outputs": [],
+      "source": [
+        "# Export requirements.txt (if needed)\n",
+        "%pip freeze > requirements.txt"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "03KLZGI_a5W5"
+      },
+      "source": [
+        "## API Keys\n",
+        "\n",
+        "Use these cells to load the API keys required for this notebook. The below code cell uses the `getpass` library."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 4,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "5smcWj4DbFgy",
+        "outputId": "969c13c3-2b77-4d7b-aa69-e24629894e6a"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "··········\n"
+          ]
+        }
+      ],
+      "source": [
+        "openai_api_key = getpass()\n",
+        "os.environ[\"OPENAI_API_KEY\"] = openai_api_key"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 35,
+      "metadata": {
+        "id": "Jgh9igPesX3F"
+      },
+      "outputs": [],
+      "source": [
+        "def splitter(text):\n",
+        "  # Split input text\n",
+        "  text_splitter = RecursiveCharacterTextSplitter(chunk_size=1500, chunk_overlap=150)\n",
+        "  splits = text_splitter.split_text(text)\n",
+        "  return splits"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "F2W_fMfRUJj2"
+      },
+      "source": [
+        "## YouTube"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 36,
+      "metadata": {
+        "id": "xm2aHrWdvztG"
+      },
+      "outputs": [],
+      "source": [
+        "def youtube_transcript(urls, save_dir = \"content\"):\n",
+        "  # Transcribe the videos to text\n",
+        "  # save_dir: directory to save audio files\n",
+        "  youtube_loader = GenericLoader(YoutubeAudioLoader(urls, save_dir), OpenAIWhisperParser())\n",
+        "  youtube_docs = youtube_loader.load()\n",
+        "  # Combine doc\n",
+        "  combined_docs = [doc.page_content for doc in youtube_docs]\n",
+        "  text = \" \".join(combined_docs)\n",
+        "  return text"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "u3pRTWrBv_oJ"
+      },
+      "outputs": [],
+      "source": [
+        "# Two Karpathy lecture videos\n",
+        "urls = [\"https://youtu.be/kCc8FmEb1nY\", \"https://youtu.be/VMj-3S1tku0\"]\n",
+        "youtube_text = youtube_transcript(urls)\n",
+        "youtube_text"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "wsMTgKjnUmql"
+      },
+      "source": [
+        "## Websites"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 25,
+      "metadata": {
+        "id": "aaIuM970pupK"
+      },
+      "outputs": [],
+      "source": [
+        "url = \"https://www.espn.com/\""
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "B2CW1oIgp5w3"
+      },
+      "source": [
+        "### WebBaseLoader"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 42,
+      "metadata": {
+        "id": "MYw1qpovlnxe"
+      },
+      "outputs": [],
+      "source": [
+        "def website_webbase(url):\n",
+        "  website_loader = WebBaseLoader(url)\n",
+        "  website_data = website_loader.load()\n",
+        "  # Combine doc\n",
+        "  combined_docs = [doc.page_content for doc in website_data]\n",
+        "  text = \" \".join(combined_docs)\n",
+        "  return text"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 43,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/",
+          "height": 139
+        },
+        "id": "M2A-lEpasbVo",
+        "outputId": "0936e800-b254-47a8-b865-ed10bec191b1"
+      },
+      "outputs": [
+        {
+          "data": {
+            "application/vnd.google.colaboratory.intrinsic+json": {
+              "type": "string"
+            },
+            "text/plain": [
+              "\"\\n\\n\\n\\n\\n\\n\\n\\n\\nESPN - Serving Sports Fans. Anytime. Anywhere.\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n        Skip to main content\\n    \\n\\n        Skip to navigation\\n    \\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n<\\n\\n>\\n\\n\\n\\n\\n\\n\\n\\n\\n\\nMenuESPN\\n\\n\\nSearch\\n\\n\\n\\nscores\\n\\n\\n\\nNFLNBANHLMLBSoccerTennis…NCAAFNCAAMNCAAWSports BettingBoxingCFLNCAACricketF1GolfHorseMMANASCARNBA G LeagueOlympic SportsPLLRacingRN BBRN FBRugbyWNBAWWEX GamesXFLMore ESPNFantasyListenWatchESPN+\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n  \\n\\nSUBSCRIBE NOW\\n\\n\\n\\n\\n\\nThe Ultimate Fighter: Season 31\\n\\n\\n\\n\\n\\n\\n\\nWimbledon: Select Courts\\n\\n\\n\\n\\n\\n\\n\\nNBA Summer League: Select Games\\n\\n\\n\\n\\n\\n\\n\\nProjecting Messi's Performance In MLS\\n\\n\\nQuick Links\\n\\n\\n\\n\\nNBA Summer League\\n\\n\\n\\n\\n\\n\\n\\nNBA Free Agency Buzz\\n\\n\\n\\n\\n\\n\\n\\nNBA Trade Machine\\n\\n\\n\\n\\n\\n\\n\\n2023 MLB Draft\\n\\n\\n\\n\\n\\n\\n\\n2023 MLB All-Star Weekend\\n\\n\\n\\n\\n\\n\\n\\nNHL Free Agency\\n\\n\\n\\n\\n\\n\\n\\nWomen's World Cup\\n\\n\\n\\n\\n\\n\\n\\nHow To Watch PGA TOUR\\n\\n\\n\\n\\n\\n\\nFavorites\\n\\n\\n\\n\\n\\n\\n      Manage Favorites\\n      \\n\\n\\n\\nCustomize ESPNSign UpLog InESPN Sites\\n\\n\\n\\n\\nESPN Deportes\\n\\n\\n\\n\\n\\n\\n\\nAndscape\\n\\n\\n\\n\\n\\n\\n\\nespnW\\n\\n\\n\\n\\n\\n\\n\\nESPNFC\\n\\n\\n\\n\\n\\n\\n\\nX Games\\n\\n\\n\\n\\n\\n\\n\\nSEC Network\\n\\n\\nESPN Apps\\n\\n\\n\\n\\nESPN\\n\\n\\n\\n\\n\\n\\n\\nESPN Fantasy\\n\\n\\nFollow ESPN\\n\\n\\n\\n\\nFacebook\\n\\n\\n\\n\\n\\n\\n\\nTwitter\\n\\n\\n\\n\\n\\n\\n\\nInstagram\\n\\n\\n\\n\\n\\n\\n\\nSnapchat\\n\\n\\n\\n\\n\\n\\n\\nYouTube\\n\\n\\n\\n\\n\\n\\n\\nThe ESPN Daily Podcast\\n\\n\\nMeet all 23 USWNT players going to the World Cup: Fun facts, insightful stats and moreFrom Alex Morgan to Megan Rapinoe, get to know everyone on the Women's World Cup roster on and off the field.7hCaitlin MurrayIllustration by ESPNNotable absentees: From Becky Sauerbrunn to Beth MeadThere are over 60 players who won't play at the Women's World Cup because of injury.11hSophie LawsonHow Mia Hamm ended up playing as a goalie at a World CupWomen's World Cup 2023: Schedule, teams, venues, moreTOP HEADLINESKamara agrees to plea deal in Vegas assault caseMLBPA wants tweak to pitch timer before playoffsNorthwestern to keep assistant coaches for 2023Surfing star Jones dies after accident with boardMessi lands in U.S. ahead of Inter Miami unveilingDiet to discipline: Zion knows he can do moreDjokovic ties Federer with 46th Slam semifinalTour-PIF talks eyed Norman exit, Tiger LIV teamWho are the NFL's best cornerbacks?MLB All-Star GameMeet the All-Star Game first-timersFrom rookies to breakouts to one guy finally getting his due, here's what to expect from MLB's most notable new All-Stars.10hJesse RogersAP Photo/Ted WarrenAll-Star Game: Predictions and much moreBaseball's best are in Seattle. Here are the matchups we want to see and what our experts think will happen.3hESPNTHE TALK OF SUMMER LEAGUEChet Holmgren, Chris Paul and more buzz from VegasOur NBA insiders have the latest from Vegas, including Holmgren's return, the in-season tournament and Paul's fit on the Warriors.7hNBA insidersPhoto by Chris Gardner/Getty ImagesNOT BUYING INTO ZION'S WORDSZion's diet comments not well received by Kendrick Perkins and Richard Jefferson49m1:47WIMBLEDON SCOREBOARDTUESDAY'S MATCHESSee AllBIG UPSET AT WIMBLEDONElina Svitolina takes down No. 1 seed Iga Swiatek5h0:55Takeaways from Swiatek's surprising loss to SvitolinaIga Swiatek came into Wimbledon as a favorite to win -- but grass once again proved to be her downfall.3hAlyssa RoenigkSURVEYING EVERY POSITIONWho are the NFL's best cornerbacks? Execs, coaches and scouts help rank 2023's top 10Who are the best corners in the NFL? Execs, coaches and scouts from around the league ranked their top 10 in our annual summer series.10hJeremy FowlerPhoto by Ethan Miller/Getty ImagesRanking the best players at every position: Execs make their picksFor the fourth straight year, we asked execs, coaches, scouts and players to name their top 10 at all positions.10hJeremy FowlerFITZGERALD OUT AT NORTHWESTERNCOLLEGE FOOTBALLRece Davis 'shocked' by Northwestern hazing claims under Fitzgerald19h3:22BIG 12 MEDIA DAYSCOLLEGE FOOTBALLLast stand for Texas and Oklahoma, newcomers to watch and expansion talkThe Big 12 will start a busy month of media days on Wednesday, July 12. Here are the biggest questions facing the conference ahead of the 2023 season.1mBill Connelly and Dave WilsonRaymond Carlin/Icon Sportswire Top HeadlinesKamara agrees to plea deal in Vegas assault caseMLBPA wants tweak to pitch timer before playoffsNorthwestern to keep assistant coaches for 2023Surfing star Jones dies after accident with boardMessi lands in U.S. ahead of Inter Miami unveilingDiet to discipline: Zion knows he can do moreDjokovic ties Federer with 46th Slam semifinalTour-PIF talks eyed Norman exit, Tiger LIV teamWho are the NFL's best cornerbacks?Favorites FantasyManage FavoritesFantasy HomeCustomize ESPNSign UpLog InICYMI0:38J-Rod puts on a show with record 41 HRs in Round 1Mariners star Julio Rodriguez electrifies the Seattle crowd with a record 41 home runs in Round 1 of the Home Run Derby. Best of ESPN+Illustration by ESPNRanking the NFL's best players at every position for 2023: Execs, coaches, scouts pick their top 10 at every positionBest corners in the NFL? Edge rushers? Linebackers? For the fourth straight year, we asked execs, coaches, scouts and players to name their top 10.AP Photo/John LocherThree takeaways from Victor Wembanyama's second gameKevin Pelton looks at what worked for Wembanyama on Sunday compared to Friday and where the 19-year-old can continue to improve.Illustration by ESPN2024 NFL mock draft: Jordan Reid's early first-round predictionsThree QBs in the top 10? A run on offensive linemen? Impact defenders galore? Here are Jordan Reid's early projections for next year's 32 first-round picks. Trending NowBettmann/Getty ImagesInside the worst team in NBA history, the 1972-73 SixersThe Philadelphia 76ers started the 1972-73 season by losing 21 of 23 games. They'd finish with the worst record in NBA history, 9-73. This is the story of what the team learned about themselves through turmoil.Ron Chenoy-USA TODAY SportsCups of coffee: Seven former NFL players remember their one and only gameTheir NFL careers lasted a single game. Seven members of a unique professional club discuss their journeys.Illustration by MASAThe FC 100 for 2023: Haaland, Mbappe lead our list of best men's soccer playersAfter a brief hiatus thanks to the winter World Cup in Qatar, ESPN presents its seventh annual ranking of the best men's players and coaches in world soccer! Welcome to FC 100. How to Watch on ESPN+(AP Photo/Koji Sasahara, File)How to watch the PGA Tour, Masters, PGA Championship and FedEx Cup playoffs on ESPN, ESPN+Here's everything you need to know about how to watch the PGA Tour, Masters, PGA Championship and FedEx Cup playoffs on ESPN and ESPN+. Sign up for FREE!Create A LeagueJoin a Public LeagueReactivate a LeaguePractice with a Mock DraftSign up to play the #1 Fantasy game!Create A LeagueJoin Public LeagueReactivateMock Draft NowSign up for FREE!Create A LeagueJoin a Public LeagueReactivate a LeaguePractice With a Mock DraftSign up for FREE!Create A LeagueJoin a Public LeaguePractice With a Mock Draft\\n\\nESPN+\\n\\n\\n\\n\\nThe Ultimate Fighter: Season 31\\n\\n\\n\\n\\n\\n\\n\\nWimbledon: Select Courts\\n\\n\\n\\n\\n\\n\\n\\nNBA Summer League: Select Games\\n\\n\\n\\n\\n\\n\\n\\nProjecting Messi's Performance In MLS\\n\\n\\nQuick Links\\n\\n\\n\\n\\nNBA Summer League\\n\\n\\n\\n\\n\\n\\n\\nNBA Free Agency Buzz\\n\\n\\n\\n\\n\\n\\n\\nNBA Trade Machine\\n\\n\\n\\n\\n\\n\\n\\n2023 MLB Draft\\n\\n\\n\\n\\n\\n\\n\\n2023 MLB All-Star Weekend\\n\\n\\n\\n\\n\\n\\n\\nNHL Free Agency\\n\\n\\n\\n\\n\\n\\n\\nWomen's World Cup\\n\\n\\n\\n\\n\\n\\n\\nHow To Watch PGA TOUR\\n\\n\\nESPN Sites\\n\\n\\n\\n\\nESPN Deportes\\n\\n\\n\\n\\n\\n\\n\\nAndscape\\n\\n\\n\\n\\n\\n\\n\\nespnW\\n\\n\\n\\n\\n\\n\\n\\nESPNFC\\n\\n\\n\\n\\n\\n\\n\\nX Games\\n\\n\\n\\n\\n\\n\\n\\nSEC Network\\n\\n\\nESPN Apps\\n\\n\\n\\n\\nESPN\\n\\n\\n\\n\\n\\n\\n\\nESPN Fantasy\\n\\n\\nFollow ESPN\\n\\n\\n\\n\\nFacebook\\n\\n\\n\\n\\n\\n\\n\\nTwitter\\n\\n\\n\\n\\n\\n\\n\\nInstagram\\n\\n\\n\\n\\n\\n\\n\\nSnapchat\\n\\n\\n\\n\\n\\n\\n\\nYouTube\\n\\n\\n\\n\\n\\n\\n\\nThe ESPN Daily Podcast\\n\\n\\nTerms of UsePrivacy PolicyYour US State Privacy RightsChildren's Online Privacy PolicyInterest-Based AdsAbout Nielsen MeasurementDo Not Sell or Share My Personal InformationContact UsDisney Ad Sales SiteWork for ESPNCopyright: © ESPN Enterprises, Inc. All rights reserved.\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\""
+            ]
+          },
+          "execution_count": 43,
+          "metadata": {},
+          "output_type": "execute_result"
+        }
+      ],
+      "source": [
+        "webbase_text = website_webbase(url)\n",
+        "webbase_text"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "fLFNd6S8oAlD"
+      },
+      "source": [
+        "### Trafilatura Parsing\n",
+        "\n",
+        "[Tralifatura](https://trafilatura.readthedocs.io/en/latest/) is a Python and command-line utility which attempts to extracts the most relevant information from a given website.  "
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 44,
+      "metadata": {
+        "id": "H3gtJjSfoK5C"
+      },
+      "outputs": [],
+      "source": [
+        "def website_trafilatura(url):\n",
+        "  downloaded = trafilatura.fetch_url(url)\n",
+        "  return trafilatura.extract(downloaded)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 45,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/",
+          "height": 52
+        },
+        "id": "ft8QDKTTsekG",
+        "outputId": "7eefb402-470d-4ff6-df72-4834a610f17a"
+      },
+      "outputs": [
+        {
+          "data": {
+            "application/vnd.google.colaboratory.intrinsic+json": {
+              "type": "string"
+            },
+            "text/plain": [
+              "'|Sports|\\n|scores||News|\\n|© 2023 ESPN Internet Ventures. Terms of Use and Privacy Policy and Safety Information / Your California Privacy Rights are applicable to you. All rights reserved.|\\n|\\nMore From ESPN:\\n|\\nESPN en Español | Andscape | FiveThirtyEight | ESPN FC | ESPNCricinfo'"
+            ]
+          },
+          "execution_count": 45,
+          "metadata": {},
+          "output_type": "execute_result"
+        }
+      ],
+      "source": [
+        "trafilatura_text = website_trafilatura(url)\n",
+        "trafilatura_text"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "evXJEtZtobn0"
+      },
+      "source": [
+        "### jusText\n",
+        "\n",
+        "[jusText](https://pypi.org/project/jusText/) is another Python library for extracting content from a website."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 46,
+      "metadata": {
+        "id": "AfahISIvph_Y"
+      },
+      "outputs": [],
+      "source": [
+        "def website_justext(url):\n",
+        "  response = requests.get(url)\n",
+        "  paragraphs = justext.justext(response.content, justext.get_stoplist(\"English\"))\n",
+        "  content = [paragraph.text for paragraph in paragraphs \\\n",
+        "            if not paragraph.is_boilerplate]\n",
+        "  text = \" \".join(content)\n",
+        "  return text"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 47,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/",
+          "height": 52
+        },
+        "id": "eB_BdMjXsg3T",
+        "outputId": "b65bfc3f-2538-4b0a-99da-926855f9e21d"
+      },
+      "outputs": [
+        {
+          "data": {
+            "application/vnd.google.colaboratory.intrinsic+json": {
+              "type": "string"
+            },
+            "text/plain": [
+              "\"Trending Now The Philadelphia 76ers started the 1972-73 season by losing 21 of 23 games. They'd finish with the worst record in NBA history, 9-73. This is the story of what the team learned about themselves through turmoil.\""
+            ]
+          },
+          "execution_count": 47,
+          "metadata": {},
+          "output_type": "execute_result"
+        }
+      ],
+      "source": [
+        "justext_text = website_justext(url)\n",
+        "justext_text"
+      ]
+    }
+  ],
+  "metadata": {
+    "colab": {
+      "include_colab_link": true,
+      "provenance": []
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}

grading_from_json.ipynb

@@ -0,0 +1,606 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "colab_type": "text",
+    "id": "view-in-github"
+   },
+   "source": [
+    "<a href=\"https://colab.research.google.com/github/vanderbilt-data-science/lo-achievement/blob/main/grading_from_json.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "kfO7rE64ZTI_"
+   },
+   "outputs": [],
+   "source": [
+    "!pip install openai"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {
+    "id": "f26sZpe-MCCj"
+   },
+   "outputs": [],
+   "source": [
+    "import json\n",
+    "import openai\n",
+    "import os\n",
+    "import pandas as pd"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 614
+    },
+    "id": "BVTr_mR0XIJI",
+    "outputId": "897e41a0-d5e1-4b5f-d254-0a6e0f6aa3fa"
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <div id=\"df-e24b7014-4d98-4fc5-9ff1-07fa5c26ba5e\">\n",
+       "    <div class=\"colab-df-container\">\n",
+       "      <div>\n",
+       "<style scoped>\n",
+       "    .dataframe tbody tr th:only-of-type {\n",
+       "        vertical-align: middle;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe tbody tr th {\n",
+       "        vertical-align: top;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead th {\n",
+       "        text-align: right;\n",
+       "    }\n",
+       "</style>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>timestamp</th>\n",
+       "      <th>author</th>\n",
+       "      <th>message</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>0</th>\n",
+       "      <td>2023-06-07 08:16:00+00:00</td>\n",
+       "      <td>assistant</td>\n",
+       "      <td>Question 1:\\nWhich of the following statements...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>1</th>\n",
+       "      <td>2023-06-07 08:16:30+00:00</td>\n",
+       "      <td>user</td>\n",
+       "      <td>C</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2</th>\n",
+       "      <td>2023-06-07 08:17:00+00:00</td>\n",
+       "      <td>assistant</td>\n",
+       "      <td>Correct! Option C is the correct answer...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>3</th>\n",
+       "      <td>2023-06-07 08:17:30+00:00</td>\n",
+       "      <td>assistant</td>\n",
+       "      <td>Question 2:\\nWhich of the following expenses a...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>4</th>\n",
+       "      <td>2023-06-07 08:18:00+00:00</td>\n",
+       "      <td>user</td>\n",
+       "      <td>A</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>5</th>\n",
+       "      <td>2023-06-07 08:18:30+00:00</td>\n",
+       "      <td>assistant</td>\n",
+       "      <td>I'm sorry, but your answer is incorrect...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>6</th>\n",
+       "      <td>2023-06-07 08:19:00+00:00</td>\n",
+       "      <td>assistant</td>\n",
+       "      <td>Question 2 (Revised):\\nWhich of the following ...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>7</th>\n",
+       "      <td>2023-06-07 08:19:30+00:00</td>\n",
+       "      <td>user</td>\n",
+       "      <td>D</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>8</th>\n",
+       "      <td>2023-06-07 08:20:00+00:00</td>\n",
+       "      <td>assistant</td>\n",
+       "      <td>Correct! Option D is the correct answer...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>9</th>\n",
+       "      <td>2023-06-07 08:20:30+00:00</td>\n",
+       "      <td>assistant</td>\n",
+       "      <td>Question 3:\\nWhat is the purpose of capitalizi...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>10</th>\n",
+       "      <td>2023-06-07 08:21:00+00:00</td>\n",
+       "      <td>user</td>\n",
+       "      <td>C</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>11</th>\n",
+       "      <td>2023-06-07 08:21:30+00:00</td>\n",
+       "      <td>assistant</td>\n",
+       "      <td>Correct! Option C is the correct answer...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>12</th>\n",
+       "      <td>2023-06-07 08:22:00+00:00</td>\n",
+       "      <td>assistant</td>\n",
+       "      <td>Question 4:\\nWhich financial statement provide...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>13</th>\n",
+       "      <td>2023-06-07 08:22:30+00:00</td>\n",
+       "      <td>user</td>\n",
+       "      <td>C</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>14</th>\n",
+       "      <td>2023-06-07 08:23:00+00:00</td>\n",
+       "      <td>assistant</td>\n",
+       "      <td>Correct! Option C is the correct answer...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>15</th>\n",
+       "      <td>2023-06-07 08:23:30+00:00</td>\n",
+       "      <td>assistant</td>\n",
+       "      <td>Question 5:\\nWhat is the purpose of the matchi...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>16</th>\n",
+       "      <td>2023-06-07 08:24:00+00:00</td>\n",
+       "      <td>user</td>\n",
+       "      <td>B</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>17</th>\n",
+       "      <td>2023-06-07 08:24:30+00:00</td>\n",
+       "      <td>assistant</td>\n",
+       "      <td>Correct! Option B is the correct answer...</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "</div>\n",
+       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-e24b7014-4d98-4fc5-9ff1-07fa5c26ba5e')\"\n",
+       "              title=\"Convert this dataframe to an interactive table.\"\n",
+       "              style=\"display:none;\">\n",
+       "        \n",
+       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
+       "       width=\"24px\">\n",
+       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
+       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
+       "  </svg>\n",
+       "      </button>\n",
+       "      \n",
+       "  <style>\n",
+       "    .colab-df-container {\n",
+       "      display:flex;\n",
+       "      flex-wrap:wrap;\n",
+       "      gap: 12px;\n",
+       "    }\n",
+       "\n",
+       "    .colab-df-convert {\n",
+       "      background-color: #E8F0FE;\n",
+       "      border: none;\n",
+       "      border-radius: 50%;\n",
+       "      cursor: pointer;\n",
+       "      display: none;\n",
+       "      fill: #1967D2;\n",
+       "      height: 32px;\n",
+       "      padding: 0 0 0 0;\n",
+       "      width: 32px;\n",
+       "    }\n",
+       "\n",
+       "    .colab-df-convert:hover {\n",
+       "      background-color: #E2EBFA;\n",
+       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
+       "      fill: #174EA6;\n",
+       "    }\n",
+       "\n",
+       "    [theme=dark] .colab-df-convert {\n",
+       "      background-color: #3B4455;\n",
+       "      fill: #D2E3FC;\n",
+       "    }\n",
+       "\n",
+       "    [theme=dark] .colab-df-convert:hover {\n",
+       "      background-color: #434B5C;\n",
+       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
+       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
+       "      fill: #FFFFFF;\n",
+       "    }\n",
+       "  </style>\n",
+       "\n",
+       "      <script>\n",
+       "        const buttonEl =\n",
+       "          document.querySelector('#df-e24b7014-4d98-4fc5-9ff1-07fa5c26ba5e button.colab-df-convert');\n",
+       "        buttonEl.style.display =\n",
+       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
+       "\n",
+       "        async function convertToInteractive(key) {\n",
+       "          const element = document.querySelector('#df-e24b7014-4d98-4fc5-9ff1-07fa5c26ba5e');\n",
+       "          const dataTable =\n",
+       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
+       "                                                     [key], {});\n",
+       "          if (!dataTable) return;\n",
+       "\n",
+       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
+       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
+       "            + ' to learn more about interactive tables.';\n",
+       "          element.innerHTML = '';\n",
+       "          dataTable['output_type'] = 'display_data';\n",
+       "          await google.colab.output.renderOutput(dataTable, element);\n",
+       "          const docLink = document.createElement('div');\n",
+       "          docLink.innerHTML = docLinkHtml;\n",
+       "          element.appendChild(docLink);\n",
+       "        }\n",
+       "      </script>\n",
+       "    </div>\n",
+       "  </div>\n",
+       "  "
+      ],
+      "text/plain": [
+       "                   timestamp     author  \\\n",
+       "0  2023-06-07 08:16:00+00:00  assistant   \n",
+       "1  2023-06-07 08:16:30+00:00       user   \n",
+       "2  2023-06-07 08:17:00+00:00  assistant   \n",
+       "3  2023-06-07 08:17:30+00:00  assistant   \n",
+       "4  2023-06-07 08:18:00+00:00       user   \n",
+       "5  2023-06-07 08:18:30+00:00  assistant   \n",
+       "6  2023-06-07 08:19:00+00:00  assistant   \n",
+       "7  2023-06-07 08:19:30+00:00       user   \n",
+       "8  2023-06-07 08:20:00+00:00  assistant   \n",
+       "9  2023-06-07 08:20:30+00:00  assistant   \n",
+       "10 2023-06-07 08:21:00+00:00       user   \n",
+       "11 2023-06-07 08:21:30+00:00  assistant   \n",
+       "12 2023-06-07 08:22:00+00:00  assistant   \n",
+       "13 2023-06-07 08:22:30+00:00       user   \n",
+       "14 2023-06-07 08:23:00+00:00  assistant   \n",
+       "15 2023-06-07 08:23:30+00:00  assistant   \n",
+       "16 2023-06-07 08:24:00+00:00       user   \n",
+       "17 2023-06-07 08:24:30+00:00  assistant   \n",
+       "\n",
+       "                                              message  \n",
+       "0   Question 1:\\nWhich of the following statements...  \n",
+       "1                                                   C  \n",
+       "2          Correct! Option C is the correct answer...  \n",
+       "3   Question 2:\\nWhich of the following expenses a...  \n",
+       "4                                                   A  \n",
+       "5          I'm sorry, but your answer is incorrect...  \n",
+       "6   Question 2 (Revised):\\nWhich of the following ...  \n",
+       "7                                                   D  \n",
+       "8          Correct! Option D is the correct answer...  \n",
+       "9   Question 3:\\nWhat is the purpose of capitalizi...  \n",
+       "10                                                  C  \n",
+       "11         Correct! Option C is the correct answer...  \n",
+       "12  Question 4:\\nWhich financial statement provide...  \n",
+       "13                                                  C  \n",
+       "14         Correct! Option C is the correct answer...  \n",
+       "15  Question 5:\\nWhat is the purpose of the matchi...  \n",
+       "16                                                  B  \n",
+       "17         Correct! Option B is the correct answer...  "
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "df = pd.read_json('demo_json.json')\n",
+    "pd.read_json('demo_json.json')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {
+    "id": "anSNlvqlXh6i"
+   },
+   "outputs": [],
+   "source": [
+    "openai.api_key = \"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\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 627
+    },
+    "id": "udujJrX6SryU",
+    "outputId": "9b182162-7c1c-4d5a-be56-16947ddcda33"
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <div id=\"df-5123f950-1dca-46a6-be4d-dab5de1f8899\">\n",
+       "    <div class=\"colab-df-container\">\n",
+       "      <div>\n",
+       "<style scoped>\n",
+       "    .dataframe tbody tr th:only-of-type {\n",
+       "        vertical-align: middle;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe tbody tr th {\n",
+       "        vertical-align: top;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead th {\n",
+       "        text-align: right;\n",
+       "    }\n",
+       "</style>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>Question</th>\n",
+       "      <th>Correct Answer</th>\n",
+       "      <th>User Answer</th>\n",
+       "      <th>Evaluation</th>\n",
+       "      <th>Score</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>0</th>\n",
+       "      <td>Question 1:\\nWhich of the following statements...</td>\n",
+       "      <td>C</td>\n",
+       "      <td>C</td>\n",
+       "      <td>correct.</td>\n",
+       "      <td>1</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>1</th>\n",
+       "      <td>Question 2 (Revised):\\nWhich of the following ...</td>\n",
+       "      <td>D</td>\n",
+       "      <td>D</td>\n",
+       "      <td>incorrect. the correct answer is d, software d...</td>\n",
+       "      <td>1</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2</th>\n",
+       "      <td>Question 3:\\nWhat is the purpose of capitalizi...</td>\n",
+       "      <td>C</td>\n",
+       "      <td>C</td>\n",
+       "      <td>incorrect. the correct answer is b.</td>\n",
+       "      <td>1</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>3</th>\n",
+       "      <td>Question 4:\\nWhich financial statement provide...</td>\n",
+       "      <td>C</td>\n",
+       "      <td>C</td>\n",
+       "      <td>correct</td>\n",
+       "      <td>2</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>4</th>\n",
+       "      <td>Question 5:\\nWhat is the purpose of the matchi...</td>\n",
+       "      <td>B</td>\n",
+       "      <td>B</td>\n",
+       "      <td>correct</td>\n",
+       "      <td>3</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "</div>\n",
+       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-5123f950-1dca-46a6-be4d-dab5de1f8899')\"\n",
+       "              title=\"Convert this dataframe to an interactive table.\"\n",
+       "              style=\"display:none;\">\n",
+       "        \n",
+       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
+       "       width=\"24px\">\n",
+       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
+       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
+       "  </svg>\n",
+       "      </button>\n",
+       "      \n",
+       "  <style>\n",
+       "    .colab-df-container {\n",
+       "      display:flex;\n",
+       "      flex-wrap:wrap;\n",
+       "      gap: 12px;\n",
+       "    }\n",
+       "\n",
+       "    .colab-df-convert {\n",
+       "      background-color: #E8F0FE;\n",
+       "      border: none;\n",
+       "      border-radius: 50%;\n",
+       "      cursor: pointer;\n",
+       "      display: none;\n",
+       "      fill: #1967D2;\n",
+       "      height: 32px;\n",
+       "      padding: 0 0 0 0;\n",
+       "      width: 32px;\n",
+       "    }\n",
+       "\n",
+       "    .colab-df-convert:hover {\n",
+       "      background-color: #E2EBFA;\n",
+       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
+       "      fill: #174EA6;\n",
+       "    }\n",
+       "\n",
+       "    [theme=dark] .colab-df-convert {\n",
+       "      background-color: #3B4455;\n",
+       "      fill: #D2E3FC;\n",
+       "    }\n",
+       "\n",
+       "    [theme=dark] .colab-df-convert:hover {\n",
+       "      background-color: #434B5C;\n",
+       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
+       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
+       "      fill: #FFFFFF;\n",
+       "    }\n",
+       "  </style>\n",
+       "\n",
+       "      <script>\n",
+       "        const buttonEl =\n",
+       "          document.querySelector('#df-5123f950-1dca-46a6-be4d-dab5de1f8899 button.colab-df-convert');\n",
+       "        buttonEl.style.display =\n",
+       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
+       "\n",
+       "        async function convertToInteractive(key) {\n",
+       "          const element = document.querySelector('#df-5123f950-1dca-46a6-be4d-dab5de1f8899');\n",
+       "          const dataTable =\n",
+       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
+       "                                                     [key], {});\n",
+       "          if (!dataTable) return;\n",
+       "\n",
+       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
+       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
+       "            + ' to learn more about interactive tables.';\n",
+       "          element.innerHTML = '';\n",
+       "          dataTable['output_type'] = 'display_data';\n",
+       "          await google.colab.output.renderOutput(dataTable, element);\n",
+       "          const docLink = document.createElement('div');\n",
+       "          docLink.innerHTML = docLinkHtml;\n",
+       "          element.appendChild(docLink);\n",
+       "        }\n",
+       "      </script>\n",
+       "    </div>\n",
+       "  </div>\n",
+       "  "
+      ],
+      "text/plain": [
+       "                                            Question Correct Answer  \\\n",
+       "0  Question 1:\\nWhich of the following statements...              C   \n",
+       "1  Question 2 (Revised):\\nWhich of the following ...              D   \n",
+       "2  Question 3:\\nWhat is the purpose of capitalizi...              C   \n",
+       "3  Question 4:\\nWhich financial statement provide...              C   \n",
+       "4  Question 5:\\nWhat is the purpose of the matchi...              B   \n",
+       "\n",
+       "  User Answer                                         Evaluation  Score  \n",
+       "0           C                                           correct.      1  \n",
+       "1           D  incorrect. the correct answer is d, software d...      1  \n",
+       "2           C                incorrect. the correct answer is b.      1  \n",
+       "3           C                                            correct      2  \n",
+       "4           B                                            correct      3  "
+      ]
+     },
+     "execution_count": 8,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "# Initialize necessary variables\n",
+    "prompt = \"\"\n",
+    "question = \"\"\n",
+    "correct_answer = \"\"\n",
+    "user_answer = \"\"\n",
+    "\n",
+    "# Initialize score\n",
+    "score = 0\n",
+    "\n",
+    "# Initialize an empty list to hold row data\n",
+    "row_data = []\n",
+    "\n",
+    "for index, row in df.iterrows():\n",
+    "    author = row['author']\n",
+    "    message = row['message']\n",
+    "\n",
+    "    # Choose the appropriate prompt based on the author\n",
+    "    if author == 'assistant':\n",
+    "        if 'Question' in message:\n",
+    "            question = message\n",
+    "            user_answer = ''  # Reset user_answer after a new question\n",
+    "        elif 'Correct! Option' in message:\n",
+    "            correct_answer = message.split('Option ')[1][0]\n",
+    "            if user_answer:  # If user_answer exists, make the API call\n",
+    "                prompt = f\"Given the following question:\\n{question}\\nThe student responded with: {user_answer}\\nIs the student's response correct or incorrect?\"\n",
+    "\n",
+    "                # Make an API call to OpenAI\n",
+    "                api_response = openai.Completion.create(\n",
+    "                    engine='text-davinci-003',\n",
+    "                    prompt=prompt,\n",
+    "                    max_tokens=100,\n",
+    "                    temperature=0.7,\n",
+    "                    n=1,\n",
+    "                    stop=None\n",
+    "                )\n",
+    "\n",
+    "                # Extract and evaluate the generated response\n",
+    "                generated_response = api_response.choices[0].text.strip().lower()\n",
+    "\n",
+    "                # Update score based on generated_response\n",
+    "                if 'correct' in generated_response and 'incorrect' not in generated_response:\n",
+    "                    score += 1\n",
+    "\n",
+    "                # Create a dictionary for the current row\n",
+    "                row_dict = {\n",
+    "                    'Question': question,\n",
+    "                    'Correct Answer': correct_answer,\n",
+    "                    'User Answer': user_answer,\n",
+    "                    'Evaluation': generated_response,\n",
+    "                    'Score': score\n",
+    "                }\n",
+    "                # Append the row dictionary to row_data\n",
+    "                row_data.append(row_dict)\n",
+    "\n",
+    "    elif author == 'user':\n",
+    "        user_answer = message\n",
+    "\n",
+    "# Create a DataFrame from row_data\n",
+    "output_df = pd.DataFrame(row_data)\n",
+    "output_df\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "colab": {
+   "authorship_tag": "ABX9TyOn+FniXzrkHNKH5uAKgyUD",
+   "include_colab_link": true,
+   "provenance": []
+  },
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.8.16"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

gradio_application.ipynb

@@ -0,0 +1,855 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "colab_type": "text",
+        "id": "view-in-github"
+      },
+      "source": [
+        "<a href=\"https://colab.research.google.com/github/vanderbilt-data-science/lo-achievement/blob/update_UI_with_prompts/gradio_application.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "x_Vp8SiKM4p1"
+      },
+      "source": [
+        "# Gradio Interface Draft\n",
+        "\n",
+        "The goal of this notebook is to show a draft of a comprehensive Gradio interface though which students can interface with an LLM and self-study."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "o_60X8H3NEne"
+      },
+      "source": [
+        "## Libraries"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 1,
+      "metadata": {
+        "id": "pxcqXgg2aAN7"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "\u001b[33mDEPRECATION: pyodbc 4.0.0-unsupported has a non-standard version number. pip 23.3 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pyodbc or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063\u001b[0m\u001b[33m\n",
+            "\u001b[0m\u001b[33mDEPRECATION: pyodbc 4.0.0-unsupported has a non-standard version number. pip 23.3 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pyodbc or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063\u001b[0m\u001b[33m\n",
+            "\u001b[0m\u001b[33mDEPRECATION: pyodbc 4.0.0-unsupported has a non-standard version number. pip 23.3 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pyodbc or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063\u001b[0m\u001b[33m\n",
+            "\u001b[0m\u001b[33mDEPRECATION: pyodbc 4.0.0-unsupported has a non-standard version number. pip 23.3 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pyodbc or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063\u001b[0m\u001b[33m\n",
+            "\u001b[0m\u001b[33mDEPRECATION: pyodbc 4.0.0-unsupported has a non-standard version number. pip 23.3 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pyodbc or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063\u001b[0m\u001b[33m\n",
+            "\u001b[0m\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
+            "anaconda-project 0.10.1 requires ruamel-yaml, which is not installed.\n",
+            "conda-repo-cli 1.0.4 requires pathlib, which is not installed.\n",
+            "daal4py 2021.3.0 requires daal==2021.2.3, which is not installed.\n",
+            "spyder 5.1.5 requires pyqt5<5.13, which is not installed.\n",
+            "spyder 5.1.5 requires pyqtwebengine<5.13, which is not installed.\n",
+            "cookiecutter 1.7.2 requires MarkupSafe<2.0.0, but you have markupsafe 2.1.3 which is incompatible.\n",
+            "numba 0.54.1 requires numpy<1.21,>=1.17, but you have numpy 1.26.0 which is incompatible.\n",
+            "pyppeteer 1.0.2 requires websockets<11.0,>=10.0, but you have websockets 11.0.3 which is incompatible.\n",
+            "scipy 1.7.1 requires numpy<1.23.0,>=1.16.5, but you have numpy 1.26.0 which is incompatible.\n",
+            "transformers 4.25.1 requires tokenizers!=0.11.3,<0.14,>=0.11.1, but you have tokenizers 0.14.0 which is incompatible.\u001b[0m\u001b[31m\n",
+            "\u001b[0m\u001b[33mDEPRECATION: pyodbc 4.0.0-unsupported has a non-standard version number. pip 23.3 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pyodbc or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063\u001b[0m\u001b[33m\n",
+            "\u001b[0mCollecting Pillow==9.0.0\n",
+            "  Downloading Pillow-9.0.0-cp39-cp39-macosx_10_10_x86_64.whl (3.0 MB)\n",
+            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━��━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.0/3.0 MB\u001b[0m \u001b[31m9.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m00:01\u001b[0m\n",
+            "\u001b[?25h\u001b[33mDEPRECATION: pyodbc 4.0.0-unsupported has a non-standard version number. pip 23.3 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pyodbc or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063\u001b[0m\u001b[33m\n",
+            "\u001b[0mInstalling collected packages: Pillow\n",
+            "  Attempting uninstall: Pillow\n",
+            "    Found existing installation: Pillow 8.4.0\n",
+            "    Uninstalling Pillow-8.4.0:\n",
+            "      Successfully uninstalled Pillow-8.4.0\n",
+            "Successfully installed Pillow-9.0.0\n",
+            "\u001b[33mDEPRECATION: pyodbc 4.0.0-unsupported has a non-standard version number. pip 23.3 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pyodbc or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063\u001b[0m\u001b[33m\n",
+            "\u001b[0m"
+          ]
+        }
+      ],
+      "source": [
+        "# install libraries here\n",
+        "# -q flag for \"quiet\" install\n",
+        "!pip install -q langchain\n",
+        "!pip install -q openai\n",
+        "!pip install -q gradio\n",
+        "!pip install -q unstructured\n",
+        "!pip install -q chromadb\n",
+        "!pip install -q tiktoken\n",
+        "!pip install Pillow==9.0.0\n",
+        "!pip install -q reportlab"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 2,
+      "metadata": {
+        "id": "pEjM1tLsMZBq"
+      },
+      "outputs": [
+        {
+          "ename": "ModuleNotFoundError",
+          "evalue": "No module named 'langchain'",
+          "output_type": "error",
+          "traceback": [
+            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
+            "\u001b[0;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
+            "Cell \u001b[0;32mIn[2], line 6\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;21;01mtime\u001b[39;00m\n\u001b[1;32m      4\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mgetpass\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m getpass\n\u001b[0;32m----> 6\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangchain\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mdocstore\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mdocument\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m Document\n\u001b[1;32m      7\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangchain\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mprompts\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m PromptTemplate\n\u001b[1;32m      8\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangchain\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mdocument_loaders\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m TextLoader\n",
+            "\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'langchain'"
+          ]
+        }
+      ],
+      "source": [
+        "# import libraries here\n",
+        "import os\n",
+        "import time\n",
+        "from getpass import getpass\n",
+        "\n",
+        "from langchain.docstore.document import Document\n",
+        "from langchain.prompts import PromptTemplate\n",
+        "from langchain.document_loaders import TextLoader\n",
+        "from langchain.indexes import VectorstoreIndexCreator\n",
+        "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
+        "from langchain.embeddings import OpenAIEmbeddings\n",
+        "\n",
+        "from langchain.document_loaders.unstructured import UnstructuredFileLoader\n",
+        "from langchain.vectorstores import Chroma\n",
+        "from langchain.chains import RetrievalQA, RetrievalQAWithSourcesChain\n",
+        "from langchain.chains.qa_with_sources import load_qa_with_sources_chain\n",
+        "#from langchain.chains import ConversationalRetrievalChain\n",
+        "\n",
+        "from langchain.llms import OpenAI\n",
+        "from langchain.chat_models import ChatOpenAI\n",
+        "\n",
+        "import gradio as gr\n",
+        "from sqlalchemy import TEXT # TODO Why is sqlalchemy imported\n",
+        "\n",
+        "import pprint\n",
+        "\n",
+        "import json\n",
+        "from google.colab import files\n",
+        "from reportlab.pdfgen.canvas import Canvas"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "n0BTyPI_srMg"
+      },
+      "source": [
+        "# Export requirements.txt (if needed)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 4,
+      "metadata": {
+        "id": "NOX639OA2pOh"
+      },
+      "outputs": [],
+      "source": [
+        "!pip freeze > requirements.txt"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "03KLZGI_a5W5"
+      },
+      "source": [
+        "## API Keys\n",
+        "\n",
+        "Use these cells to load the API keys required for this notebook. The below code cell uses the `getpass` library."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 3,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "5smcWj4DbFgy",
+        "outputId": "0aec236a-dfc8-4afb-d97c-ff324b85bb70"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "··········\n"
+          ]
+        }
+      ],
+      "source": [
+        "openai_api_key = getpass()\n",
+        "os.environ[\"OPENAI_API_KEY\"] = openai_api_key"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 4,
+      "metadata": {
+        "id": "fck1RVxD8xSX"
+      },
+      "outputs": [],
+      "source": [
+        "llm = ChatOpenAI(model_name = 'gpt-3.5-turbo-16k')\n",
+        "# TODO OpenAI() or ChatOpenAI()?"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "UzaYNFFT4AwX"
+      },
+      "source": [
+        "# Interface"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 5,
+      "metadata": {
+        "id": "N9i8zsMmbcd3"
+      },
+      "outputs": [],
+      "source": [
+        "global db # Document-storing object (vector store / index)\n",
+        "global qa # Question-answer object; retrieves from `db`\n",
+        "global srcs # List of source documents fragments referenced by vector store\n",
+        "num_sources = 100 # Maximum number of source documents which can be shown\n",
+        "\n",
+        "srcs = []\n",
+        "# See https://github.com/hwchase17/langchain/discussions/3786 for discussion\n",
+        "# of which splitter to use\n",
+        "text_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=0)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 6,
+      "metadata": {
+        "id": "fmFg2_iyvRKI"
+      },
+      "outputs": [],
+      "source": [
+        "### Source Display ###\n",
+        "\n",
+        "def format_source_reference(document, index):\n",
+        "  \"\"\"Return a HTML element which contains the `document` info and can be\n",
+        "  referenced by the `index`.\"\"\"\n",
+        "  if 'source' in document.metadata:\n",
+        "    source_filepath, source_name = os.path.split(document.metadata['source'])\n",
+        "  else:\n",
+        "    source_name = \"text box\"\n",
+        "  return  f\"<p id='source{index}'>[{index+1}] <em>{source_name}</em><br>...{document.page_content}...</p>\"\n",
+        "\n",
+        "def format_source_citation(index):\n",
+        "  \"\"\"Return a Markdown element which references the element with the given\n",
+        "  `index`.\"\"\"\n",
+        "  return f\"[{index+1}](#source{index})\"\n",
+        "\n",
+        "def update_source_list(added_sources):\n",
+        "  \"\"\"Add any new sources to the list `srcs`; return a list of integers\n",
+        "  containing the index references for each of the given sources in `sources`.\"\"\"\n",
+        "  source_indices = []\n",
+        "  for source in added_sources:\n",
+        "    if source not in srcs: srcs.append(source)\n",
+        "    source_indices.append(srcs.index(source))\n",
+        "  return source_indices\n",
+        "\n",
+        "def get_source_display_updates(sources=srcs):\n",
+        "  \"\"\"Dynamically update the output for the given list of components to show\n",
+        "  the items as contained within the documents in `sources`.\n",
+        "  See https://github.com/gradio-app/gradio/issues/2066 for what I'm copying.\"\"\"\n",
+        "  # TODO Currently displays the first 100 sources only, which could be an issue\n",
+        "  # TODO Display an alert when the source display is being cut off (over 100\n",
+        "  # sources are stored)\n",
+        "  update_list = []\n",
+        "  for i in range(num_sources):\n",
+        "    if i < len(sources):\n",
+        "      source_text = format_source_reference(sources[i], i)\n",
+        "      update_list.append(gr.update(visible=True, value=source_text))\n",
+        "    else:\n",
+        "      update_list.append(gr.update(visible=False))\n",
+        "  return update_list"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 7,
+      "metadata": {
+        "id": "2HnXKhgPccPV"
+      },
+      "outputs": [],
+      "source": [
+        "def embed_key(openai_api_key):\n",
+        "  os.environ[\"OPENAI_API_KEY\"] = openai_api_key\n",
+        "\n",
+        "### Helper Functions ###\n",
+        "\n",
+        "def create_vector_store_from_document_segments(document_segments):\n",
+        "  \"\"\"Updates the global `db` and `qa` variables to contain the given sections\n",
+        "  of text (generated from the create_vector_store_from_file and\n",
+        "  create_vector_store_from_text_input functions).\"\"\"\n",
+        "  global db, qa\n",
+        "\n",
+        "  # TODO: Add useful metadata that indicates from *where* in the document\n",
+        "  # the segment is pulled (ideally, show what page it's from, if possible)\n",
+        "  # This requires changing how the info is displayed in the UI, but it\n",
+        "  # shouldn't be a big change\n",
+        "  embeddings = OpenAIEmbeddings()\n",
+        "  db = Chroma.from_documents(document_segments, embeddings)\n",
+        "  retriever = db.as_retriever()\n",
+        "\n",
+        "  qa = RetrievalQAWithSourcesChain.from_chain_type(llm=llm,\n",
+        "                                  chain_type=\"stuff\",\n",
+        "                                  retriever=retriever,\n",
+        "                                  return_source_documents=True)\n",
+        "\n",
+        "def get_document_segments_from_files(files):\n",
+        "  \"\"\"Returns a list of document segments taken from all of the files in the\n",
+        "  given list.\"\"\"\n",
+        "  # TODO Modify the metadata for each document to specify the location; see\n",
+        "  # colab.research.google.com/drive/142CCh-Mh1AdodIGD7PyL86slFLOcVVeI\n",
+        "  all_document_segments = []\n",
+        "  for file in files:\n",
+        "    loader = UnstructuredFileLoader(file.name)\n",
+        "    documents = loader.load()\n",
+        "    document_segments = text_splitter.split_documents(documents)\n",
+        "    all_document_segments.extend(document_segments)\n",
+        "  return all_document_segments\n",
+        "\n",
+        "def get_document_segments_from_text(text):\n",
+        "  \"\"\"Returns a list of document segments taken from the text.\"\"\"\n",
+        "  return [Document(page_content=page, metadata={\"source\":\"text box\"})\n",
+        "                      for page in text_splitter.split_text(text)]\n",
+        "\n",
+        "def create_vector_store_from_file(file):\n",
+        "  \"\"\"Updates the global `db` and `qa` variables to contain the given file.\"\"\"\n",
+        "  # TODO change the code to allow input of multiple files (see previous\n",
+        "  # implementation of this in some other notebook I made)\n",
+        "  loader = UnstructuredFileLoader(file.name)\n",
+        "  documents = loader.load()\n",
+        "  document_segments = text_splitter.split_documents(documents)\n",
+        "  create_vector_store_from_document_segments(document_segments)\n",
+        "\n",
+        "def create_vector_store_from_files(files):\n",
+        "  \"\"\"Updates the global `db` and `qa` variables to contain the given files.\"\"\"\n",
+        "  document_segments = get_document_segments_from_files(files)\n",
+        "  create_vector_store_from_document_segments(document_segments)\n",
+        "\n",
+        "def create_vector_store_from_text(text):\n",
+        "  \"\"\"Updates the global `db` and `qa` variables to contain the given string.\"\"\"\n",
+        "  document_segments = get_document_segments_from_text(text)\n",
+        "  create_vector_store_from_document_segments(document_segments)\n",
+        "\n",
+        "# TODO I keep on getting the following error (or something similar):\n",
+        "# openai.error.InvalidRequestError: This model's maximum context length is 4097\n",
+        "# tokens, however you requested 4333 tokens (4077 in your prompt; 256 for the\n",
+        "# completion). Please reduce your prompt; or completion length.\n",
+        "\n",
+        "def construct_prompt(prompt, learning_objective):\n",
+        "  # TODO Use prompt templates instead of just adding the strings together\n",
+        "  return prompt + \" \" + learning_objective\n",
+        "\n",
+        "def generate_response(prompt, learning_objective=''):\n",
+        "  \"\"\"Get the LLM response to the given prompt with the given learning objective,\n",
+        "  referencing the global `qa` variable.\"\"\"\n",
+        "  global db, qa\n",
+        "  qa_prompt = construct_prompt(prompt, learning_objective)\n",
+        "  result = qa({\"question\": qa_prompt})\n",
+        "  return result[\"answer\"], result[\"source_documents\"]\n",
+        "\n",
+        "def generate_response_with_document_update(prompt, learning_objective=''):\n",
+        "  \"\"\"Get the LLM response to the given prompt with the given learning objective,\n",
+        "  referencing the global `qa` variable, and also return the document updates.\"\"\"\n",
+        "  answer, sources = generate_response(prompt, learning_objective)\n",
+        "  update_source_list(sources)\n",
+        "  return [answer, *get_source_display_updates()]"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 8,
+      "metadata": {
+        "id": "HMQcRCGjY_7p"
+      },
+      "outputs": [],
+      "source": [
+        "### Source Display ###\n",
+        "\n",
+        "def format_source_reference(document, index):\n",
+        "  \"\"\"Return a HTML element which contains the `document` info and can be\n",
+        "  referenced by the `index`.\"\"\"\n",
+        "  source_filepath, source_filename = os.path.split(document.metadata['source'])\n",
+        "  return  f\"<p id='source{index}'>[{index+1}] <em>{source_filename}</em><br>...{document.page_content}...</p>\"\n",
+        "\n",
+        "def format_source_citation(index):\n",
+        "  \"\"\"Return a Markdown element which references the element with the given\n",
+        "  `index`.\"\"\"\n",
+        "  return f\"[{index+1}](#source{index})\"\n",
+        "\n",
+        "def update_source_list(added_sources):\n",
+        "  \"\"\"Add any new sources to the list `srcs`; return a list of integers\n",
+        "  containing the index references for each of the given sources in `sources`.\"\"\"\n",
+        "  source_indices = []\n",
+        "  for source in added_sources:\n",
+        "    if source not in srcs: srcs.append(source)\n",
+        "    source_indices.append(srcs.index(source))\n",
+        "  return source_indices\n",
+        "\n",
+        "def get_source_display_updates(sources=srcs):\n",
+        "  \"\"\"Dynamically update the output for the given list of components to show\n",
+        "  the items as contained within the documents in `sources`.\n",
+        "  See https://github.com/gradio-app/gradio/issues/2066 for what I'm copying.\"\"\"\n",
+        "  # TODO Currently displays the first 100 sources only, which could be an issue\n",
+        "  # TODO Display an alert when the source display is being cut off (over 100\n",
+        "  # sources are stored)\n",
+        "  update_list = []\n",
+        "  for i in range(num_sources):\n",
+        "    if i < len(sources):\n",
+        "      source_text = format_source_reference(sources[i], i)\n",
+        "      update_list.append(gr.update(visible=True, value=source_text))\n",
+        "    else:\n",
+        "      update_list.append(gr.update(visible=False))\n",
+        "  return update_list"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 21,
+      "metadata": {
+        "id": "XwIHEgnmQ7_q"
+      },
+      "outputs": [],
+      "source": [
+        "### Gradio Called Functions ###\n",
+        "\n",
+        "def prompt_select(selection, number, length):\n",
+        "  if selection == \"Random\":\n",
+        "    prompt = f\"Please design a {number} question quiz based on the context provided and the inputted learning objectives (if applicable). The types of questions should be randomized (including multiple choice, short answer, true/false, short answer, etc.).  Provide one question at a time, and wait for my response before providing me with feedback. Again, while the quiz may ask for multiple questions, you should only provide 1 question in you initial response. Do not include the answer in your response. If I get an answer wrong, provide me with an explanation of why it was incorrect, and then give me additional chances to respond until I get the correct choice. Explain why the correct choice is right.\"\n",
+        "  elif selection == \"Fill in the Blank\":\n",
+        "    prompt = f\"Create a {number} question fill in the blank quiz refrencing the context provided. The quiz should reflect the learning objectives (if inputted). The 'blank' part of the question should appear as '________'. The answers should reflect what word(s) should go in the blank an accurate statement. An example is the follow: 'The author of the article is ______.' The question should be a statement. Provide one question at a time, and wait for my response before providing me with feedback. Again, while the quiz may ask for multiple questions, you should only provide ONE question in you initial response. Do not include the answer in your response. If I get an answer wrong, provide me with an explanation of why it was incorrect,and then give me additional chances to respond until I get the correct choice. Explain why the correct choice is right.\"\n",
+        "  elif selection == \"Short Answer\":\n",
+        "    prompt = f\"Please design a {number} question quiz about which reflects the learning objectives (if inputted).  The questions should be short answer. Expect the correct answers to be {length} sentences long. Provide one question at a time, and wait for my response before providing me with feedback. Again, while the quiz may ask for multiple questions, you should only provide ONE question in you initial response. Do not include the answer in your response. If I get an answer wrong, provide me with an explanation of why it was incorrect, and then give me additional chances to respond until I get the correct choice. Explain why the correct answer is right.\"\n",
+        "  else:\n",
+        "    prompt = f\"Please design a {number} question {selection.lower()} quiz based on the context provided and the inputted learning objectives (if applicable). Provide one question at a time, and wait for my response before providing me with feedback. Again, while the quiz may ask for multiple questions, you should only provide 1 question in you initial response. Do not include the answer in your response. If I get an answer wrong, provide me with an explanation of why it was incorrect, and then give me additional chances to respond until I get the correct choice. Explain why the correct choice is right.\"\n",
+        "  return prompt\n",
+        "\n",
+        "'''\n",
+        "def generate_response_with_document(file_input, text_input,\n",
+        "                                    prompt, learning_objective):\n",
+        "  \"\"\"Determine which type of document input is given and call the corresponding\n",
+        "  function to generate and return a response, along with the sources.\"\"\"\n",
+        "  if file_input:\n",
+        "    return generate_response_from_files(file_input, prompt, learning_objective)\n",
+        "  elif text_input:\n",
+        "    return generate_response_from_text(text_input, prompt, learning_objective)\n",
+        "  else: # No document input\n",
+        "    # TODO add a UI indicator to the student if they forgot to provide document\n",
+        "    raise Exception(\"No source text or file given\")\n",
+        "'''\n",
+        "pass"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 10,
+      "metadata": {
+        "id": "mqMZlP4CcfNa"
+      },
+      "outputs": [],
+      "source": [
+        "### Chatbot Functions ###\n",
+        "\n",
+        "def user(user_message, history):\n",
+        "  \"\"\"Display user message and update chat history to include it.\n",
+        "  Also disables user text input until bot is finished (call to reenable_chat())\n",
+        "  See https://gradio.app/creating-a-chatbot/\"\"\"\n",
+        "  return gr.update(value=\"\", interactive=True), history + [[user_message, None]]\n",
+        "\n",
+        "def get_chatbot_response(history):\n",
+        "  \"\"\"Given the history as a list of tuples of strings, output the new history\n",
+        "  response (referencing the document via vector store `qa`), along with the\n",
+        "  currently referenced sources.\"\"\"\n",
+        "\n",
+        "  # TODO Disable input when the user has not added a document for context\n",
+        "  # Right now, if you try to use this before pressing one of the other\n",
+        "  # generator buttons, it breaks!\n",
+        "  if not qa: raise Exception(\"No source text or file given\")\n",
+        "  history = history or [] # Account for issue if history is empty\n",
+        "  # Turn nested list into string:\n",
+        "  # [[\"Hello HAL\", \"Hi Dave\"], [\"Open the pod bay doors\", \"No.\"]] ->\n",
+        "  # \"Hello HAL\\nHi Dave\\nOpen the pod bay doors\\nNo.\"\n",
+        "  # TODO: This conversion doesn't distinguish between the input of the user\n",
+        "  # and the output from the model... I could add tags for each section, e.g.\n",
+        "  # \"USER:Hello HAL\\nBOT:Hi Dave\\nUSER:Open the pod bay doors\\nBOT:No.\"\n",
+        "  # But that uses up extra tokens... we need to figure out if that's worth it\n",
+        "  history_flat = [message for exchange in history\n",
+        "               for message in exchange if message]\n",
+        "  history_string = '\\n'.join(history_flat)\n",
+        "  response = generate_response(history_string)\n",
+        "  return response\n",
+        "\n",
+        "def bot(history, use_model=True, character_crawl=False):\n",
+        "  \"\"\"Get model response based on chat history (including user input).\n",
+        "  If use_model is False, then return a placeholder value.\n",
+        "  If character_crawl, then show each character in the output one at a time.\n",
+        "  See https://gradio.app/creating-a-chatbot/\"\"\"\n",
+        "\n",
+        "  answer, sources = get_chatbot_response(history) if use_model \\\n",
+        "      else (\"This is a placeholder message.\", [])\n",
+        "\n",
+        "  # Add the link to the sources\n",
+        "  #bot_message = f\"{answer}\\n[Sources](#sources)\"\n",
+        "  source_indices = update_source_list(sources)\n",
+        "  source_citations = \" \".join(format_source_citation(source_index)\n",
+        "                      for source_index in source_indices)\n",
+        "  bot_message = answer + \"\\nSources: \" + source_citations\n",
+        "\n",
+        "  \"\"\"\n",
+        "  if character_crawl:\n",
+        "    # Display generated response, one character at a time\n",
+        "    # TODO doesn't work if the history is empty (no messages)\n",
+        "    # However, get_chatbot_response() should always output something, so it's fine\n",
+        "    history[-1][1] = \"\"\n",
+        "    for character in bot_message:\n",
+        "        history[-1][1] += character\n",
+        "        # TODO Remove timeout\n",
+        "        time.sleep(0.01)\n",
+        "        yield history\n",
+        "  else:\n",
+        "    history[-1][1] = bot_message\n",
+        "    yield history\n",
+        "  \"\"\"\n",
+        "  history[-1][1] = bot_message\n",
+        "  yield [history, *get_source_display_updates()]\n",
+        "\n",
+        "def reenable_chat():\n",
+        "  \"\"\"Called after user() disables chat and bot provides response, meaning\n",
+        "  the user is now free to add follow-up questions / comments to chatbot.\"\"\"\n",
+        "  gr.update(interactive=True)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 11,
+      "metadata": {
+        "id": "lOA1Vn9GKPeN"
+      },
+      "outputs": [],
+      "source": [
+        "# Downloadable formats\n",
+        "\n",
+        "def save_chatbot_dialogue(chatbot, filename='chatbot_dialogue.json'):\n",
+        "  with open(filename, 'w') as file:\n",
+        "    json.dump(chatbot, file)\n",
+        "  files.download(filename)\n",
+        "\n",
+        "def list_to_str(chatbot):\n",
+        "  total_interaction = \"\"\n",
+        "  for line in chatbot:\n",
+        "    if line[0]:\n",
+        "      user_message = \"User: \" + line[0] + \"\\n\"\n",
+        "      total_interaction += user_message\n",
+        "    if line[1]:\n",
+        "      chat_message = \"Bot: \" + line[1] + \"\\n\"\n",
+        "      total_interaction += chat_message\n",
+        "  return total_interaction\n",
+        "\n",
+        "from textwrap import wrap\n",
+        "def save_chatbot_dialogue_pdf(chatbot, filename=\"chatbot_dialogue.pdf\"):\n",
+        "  chatbot = list_to_str(chatbot)\n",
+        "  canvas = Canvas(filename)\n",
+        "  t = canvas.beginText()\n",
+        "  t.setFont('Helvetica', 12)\n",
+        "  t.setCharSpace(2)\n",
+        "  t.setTextOrigin(50, 750)\n",
+        "  t.textLines(chatbot)\n",
+        "  canvas.drawText(t)\n",
+        "  canvas.save()\n",
+        "  files.download(filename)\n",
+        "\n",
+        "def save_as_txt(chatbot, filename = 'chat_dialogue.txt'):\n",
+        "  chatbot = list_to_str(chatbot)\n",
+        "  with open(filename, 'w') as file:\n",
+        "    file.write(chatbot)\n",
+        "  files.download(filename)\n",
+        "\n",
+        "def save_as_csv(chatbot, filename = 'chat_dialogue.csv'):\n",
+        "  chatbot = list_to_str(chatbot)\n",
+        "  with open(filename, 'w') as file:\n",
+        "    file.write(chatbot)\n",
+        "  files.download(filename)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 12,
+      "metadata": {
+        "id": "vV9IY1H4H7UK"
+      },
+      "outputs": [],
+      "source": [
+        "# Function to save prompts (premade or custom) and return in the user input box in the chatbot\n",
+        "\n",
+        "saved_text = \"\"\n",
+        "def save_text(text):\n",
+        "    global saved_text\n",
+        "    saved_text = text\n",
+        "\n",
+        "def return_text():\n",
+        "    # Return the saved text\n",
+        "    return saved_text"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 22,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/",
+          "height": 684
+        },
+        "id": "Z6d6m-sBwnkA",
+        "outputId": "37bd4262-7eed-49aa-e064-7f0443ed9a69"
+      },
+      "outputs": [
+        {
+          "name": "stderr",
+          "output_type": "stream",
+          "text": [
+            "<ipython-input-22-8f58a957929c>:113: GradioUnusedKwargWarning: You have unused kwarg parameters in Box, please remove them: {'scale': 1}\n",
+            "  with gr.Box(elem_id=\"sources-container\", scale=1):\n"
+          ]
+        },
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "Setting queue=True in a Colab notebook requires sharing enabled. Setting `share=True` (you can turn this off by setting `share=False` in `launch()` explicitly).\n",
+            "\n",
+            "Colab notebook detected. To show errors in colab notebook, set debug=True in launch()\n",
+            "Running on public URL: https://f880b30167f96bbba1.gradio.live\n",
+            "\n",
+            "This share link expires in 72 hours. For free permanent hosting and GPU upgrades, run `gradio deploy` from Terminal to deploy to Spaces (https://huggingface.co/spaces)\n"
+          ]
+        },
+        {
+          "data": {
+            "text/html": [
+              "<div><iframe src=\"https://f880b30167f96bbba1.gradio.live\" width=\"100%\" height=\"500\" allow=\"autoplay; camera; microphone; clipboard-read; clipboard-write;\" frameborder=\"0\" allowfullscreen></iframe></div>"
+            ],
+            "text/plain": [
+              "<IPython.core.display.HTML object>"
+            ]
+          },
+          "metadata": {},
+          "output_type": "display_data"
+        },
+        {
+          "data": {
+            "text/plain": []
+          },
+          "execution_count": 22,
+          "metadata": {},
+          "output_type": "execute_result"
+        }
+      ],
+      "source": [
+        "# See https://gradio.app/custom-CSS-and-JS/\n",
+        "css=\"\"\"\n",
+        "#sources-container {\n",
+        "  overflow: scroll !important; /* Needs to override default formatting */\n",
+        "  /*max-height: 20em; */ /* Arbitrary value */\n",
+        "}\n",
+        "#sources-container > div { padding-bottom: 1em !important; /* Arbitrary value */ }\n",
+        ".short-height > * > * { min-height: 0 !important; }\n",
+        ".translucent { opacity: 0.5; }\n",
+        ".textbox_label { padding-bottom: .5em; }\n",
+        "\"\"\"\n",
+        "#srcs = [] # Reset sources (db and qa are kept the same for ease of testing)\n",
+        "\n",
+        "with gr.Blocks(css=css, analytics_enabled=False) as demo:\n",
+        "  gr.Markdown(\"# Student Notebook Thingy\")\n",
+        "  with gr.Box():\n",
+        "    gr.HTML(\"\"\"<span>Embed your OpenAI API key below; if you haven't created one already, visit\n",
+        "      <a href=\"https://platform.openai.com/account/api-keys\">platform.openai.com/account/api-keys</a>\n",
+        "    to sign up for an account and get your personal API key</span>\"\"\",\n",
+        "            elem_classes=\"textbox_label\")\n",
+        "    # TODO Find a better way to fix minor spacing issue below this element\n",
+        "    input = gr.Textbox(show_label=False, type=\"password\", container=False,\n",
+        "                      placeholder=\"●●●●●●●●●●●●●●●●●\")\n",
+        "    input.change(fn=embed_key, inputs=input, outputs=None)\n",
+        "\n",
+        "\n",
+        "  with gr.Blocks():\n",
+        "    gr.Markdown(\"## Add Document Source (File / Text / VectorStore)\")\n",
+        "    # TODO Add entry for path to vector store (should be disabled for now)\n",
+        "    with gr.Row(equal_height=True):\n",
+        "      text_input = gr.TextArea(label='Copy and paste your text below',\n",
+        "                                lines=2)\n",
+        "      # TODO This is called every time the text changes, which is very\n",
+        "      # wasteful... the solution would be to only call this when necessary,\n",
+        "      # (i.e. when generating text), but that would require some extra\n",
+        "      # parameters passed in to the generating functions... maybe\n",
+        "      # add a check for that in the functions which reference the doc store?\n",
+        "      text_input.input(create_vector_store_from_text, [text_input], None)\n",
+        "\n",
+        "      file_input = gr.Files(label=\"Load a .txt or .pdf file\",\n",
+        "                            file_types=['.pdf', '.txt'], type=\"file\",\n",
+        "                            elem_classes=\"short-height\")\n",
+        "      file_input.change(create_vector_store_from_files, [file_input], None)\n",
+        "\n",
+        "      text_input = gr.TextArea(label='Enter vector store URL, if given by instructor (WIP)',\n",
+        "                                lines=2, interactive=\n",
+        "                               False, elem_classes=\"translucent\")\n",
+        "\n",
+        "  with gr.Blocks():\n",
+        "    #gr.Markdown(\"## Optional: Enter Your Learning Objectives\")\n",
+        "    learning_objectives = gr.Textbox(label='If provided by your instructor, please input your learning objectives for this session', value='')\n",
+        "\n",
+        "  # Premade question prompts\n",
+        "  # TODO rename section?`\n",
+        "  with gr.Blocks():\n",
+        "    gr.Markdown(\"\"\"\n",
+        "    ## Generate a Premade Prompt\n",
+        "    Select your type and number of desired questions. Click \"Generate Prompt\" to get your premade prompt,\n",
+        "    and then \"Insert Prompt into Chat\" to copy the text into the chat interface below. \\\n",
+        "    You can also copy the prompt using the icon in the upper right corner and paste directly into the input box when interacting with the model.\n",
+        "    \"\"\")\n",
+        "    with gr.Row():\n",
+        "      with gr.Column():\n",
+        "        question_type = gr.Dropdown([\"Multiple Choice\", \"True or False\", \"Short Answer\", \"Fill in the Blank\", \"Random\"], label=\"Question Type\")\n",
+        "        number_of_questions = gr.Textbox(label=\"Enter desired number of questions\")\n",
+        "        sa_desired_length = gr.Dropdown([\"1-2\", \"3-4\", \"5-6\", \"6 or more\"], label = \"For short answer questions only, choose the desired sentence length for answers. The default value is 1-2 sentences.\")\n",
+        "      with gr.Column():\n",
+        "        prompt_button = gr.Button(\"Generate Prompt\")\n",
+        "        premade_prompt_output = gr.Textbox(label=\"Generated prompt (save or copy)\", show_copy_button=True)\n",
+        "        prompt_button.click(prompt_select,\n",
+        "                        inputs=[question_type, number_of_questions, sa_desired_length],\n",
+        "                        outputs=premade_prompt_output)\n",
+        "\n",
+        "    insert_premade_prompt_button = gr.Button(\"Input Prompt into Chat\")\n",
+        "\n",
+        "  '''\n",
+        "  with gr.Blocks():\n",
+        "      # TODO Consider if we want to have this section\n",
+        "      # Maybe use this section for prompt refinement? E.g. you enter prompt\n",
+        "      # and model tells you how to improve it?\n",
+        "      gr.Markdown(\"\"\"\n",
+        "      ## Write Your Own Custom Prompt\n",
+        "      For a comprehensive list of example prompts, see our repository's wiki: https://github.com/vanderbilt-data-science/lo-achievement/wiki/High-Impact-Prompts.\n",
+        "      After writing your prompt, be sure to click \"Save Prompt\" to save your prompt for chatting with the model below. You can also copy the prompt using the icon in the upper right corner and paste directly into the input box when interacting with the model.\n",
+        "      \"\"\")\n",
+        "      custom_prompt_block = gr.TextArea(label=\"Enter your own custom prompt\", show_copy_button=True)\n",
+        "      #save_prompt_block = gr.Button(\"Save Prompt\")\n",
+        "      #save_prompt_block.click(save_text, inputs=custom_prompt_block, outputs=None)\n",
+        "  '''\n",
+        "\n",
+        "  # Chatbot (https://gradio.app/creating-a-chatbot/)\n",
+        "  '''\n",
+        "  with gr.Blocks():\n",
+        "    gr.Markdown(\"\"\"\n",
+        "    ## Chat with the Model\n",
+        "    Click \"Display Prompt\" to display the premade or custom prompt that you created earlier. Then, continue chatting with the model.\n",
+        "    \"\"\")\n",
+        "    with gr.Row():\n",
+        "      show_prompt_block = gr.Button(\"Display Prompt\")\n",
+        "  '''\n",
+        "  gr.Markdown(\"## Chat with the Model\")\n",
+        "  with gr.Row(equal_height=True):\n",
+        "    with gr.Column(scale=2):\n",
+        "      chatbot = gr.Chatbot()\n",
+        "      with gr.Row():\n",
+        "        user_chat_input = gr.Textbox(label=\"User input\", scale=9)\n",
+        "        user_chat_input.submit(return_text, inputs=None, outputs=user_chat_input)\n",
+        "        user_chat_submit = gr.Button(\"Ask/answer model\", scale=1)\n",
+        "        #show_prompt_block.click(return_text, inputs=None, outputs=user_chat_input)\n",
+        "\n",
+        "    # TODO Move the sources so it's displayed to the right of the chat bot,\n",
+        "    # with the sources taking up about 1/3rd of the horizontal space\n",
+        "    with gr.Box(elem_id=\"sources-container\", scale=1):\n",
+        "      # TODO: Display document sources in a nicer format?\n",
+        "      gr.HTML(value=\"<h3 id='sources'>Sources</h3>\")\n",
+        "      sources_output = []\n",
+        "      for i in range(num_sources):\n",
+        "        source_elem = gr.HTML(visible=False)\n",
+        "        sources_output.append(source_elem)\n",
+        "\n",
+        "  # Copy text from premade prompt output to user chat input\n",
+        "  insert_premade_prompt_button.click(lambda text: text,\n",
+        "                        inputs=premade_prompt_output,\n",
+        "                        outputs=user_chat_input)\n",
+        "\n",
+        "  # Display input and output in three-ish parts\n",
+        "  # (using asynchronous functions):\n",
+        "  # First show user input, then show model output when complete\n",
+        "  # Then wait until the bot provides response and return the result\n",
+        "  # Finally, allow the user to ask a new question by reenabling input\n",
+        "  async_response = user_chat_submit.click(user,\n",
+        "            [user_chat_input, chatbot],\n",
+        "            [user_chat_input, chatbot], queue=False) \\\n",
+        "  .then(bot, chatbot, [chatbot, *sources_output], queue=True) \\\n",
+        "  .then(reenable_chat, None, [user_chat_input], queue=False)\n",
+        "\n",
+        "  with gr.Blocks():\n",
+        "    gr.Markdown(\"\"\"\n",
+        "    ## Export Your Chat History\n",
+        "    Export your chat history as a .json, PDF file, .txt, or .csv file\n",
+        "    \"\"\")\n",
+        "    with gr.Row():\n",
+        "      export_dialogue_button = gr.Button(\"JSON\")\n",
+        "      export_dialogue_button.click(save_chatbot_dialogue, chatbot, None)\n",
+        "      export_dialogue_button_pdf = gr.Button(\"PDF\")\n",
+        "      export_dialogue_button_pdf.click(save_chatbot_dialogue_pdf, chatbot, None)\n",
+        "      export_dialogue_button_txt = gr.Button(\"TXT\")\n",
+        "      export_dialogue_button_txt.click(save_as_txt, chatbot, None)\n",
+        "      export_dialogue_button_csv = gr.Button(\"CSV\")\n",
+        "      export_dialogue_button_csv.click(save_as_csv, chatbot, None)\n",
+        "\n",
+        "demo.queue()\n",
+        "#demo.launch(debug=True)\n",
+        "demo.launch()"
+      ]
+    }
+  ],
+  "metadata": {
+    "colab": {
+      "include_colab_link": true,
+      "provenance": []
+    },
+    "kernelspec": {
+      "display_name": "Python 3.11.5 64-bit",
+      "language": "python",
+      "name": "python3"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.11.5"
+    },
+    "vscode": {
+      "interpreter": {
+        "hash": "b0fa6594d8f4cbf19f97940f81e996739fb7646882a419484c72d19e05852a7e"
+      }
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}

instructor_intr_notebook.ipynb

@@ -0,0 +1,3153 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "collapsed": true,
+    "id": "brzvVeAsYiG2"
+   },
+   "source": [
+    "<a href=\"https://colab.research.google.com/github/vanderbilt-data-science/lo-achievement/blob/main/instructor_intr_notebook.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "WMKrKfx8_3fc"
+   },
+   "source": [
+    "# Instructor Grading and Assessment\n",
+    "This notebook executes grading of student submissions of chats with ChatGPT, exported in JSON. Run each cell should be run in order, and follow the prompts displayed when appropriate."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 35,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 16
+    },
+    "id": "696FqPrTYiG3",
+    "outputId": "9679a415-8ab7-4c5f-e715-954d6801b6ec"
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "import ipywidgets as widgets\n",
+    "from IPython.display import display, HTML, clear_output\n",
+    "import io\n",
+    "import zipfile\n",
+    "import os\n",
+    "import json\n",
+    "import pandas as pd\n",
+    "import glob\n",
+    "from getpass import getpass"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 36,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 16
+    },
+    "id": "fTlnrMwmYiG4",
+    "outputId": "e811e000-e9ec-43b6-d136-59d5134adeaf"
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "# \"global\" variables modified by mutability\n",
+    "grade_settings = {'learning_objectives':None,\n",
+    "                  'json_file_path':None,\n",
+    "                  'json_files':None }"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "jb0jnIE14Vuh"
+   },
+   "source": [
+    "The `InstructorGradingConfig` holds the contents of the instantiated object including making graindg settings, extracting files from a zip archive, loading JSON files into DataFrames, and displaying relevant information in the output widget."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 37,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 16
+    },
+    "id": "mPLdaWiuYiG4",
+    "outputId": "7a698bc1-7954-44ac-83c8-71d1dc410749"
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "class InstructorGradingConfig:\n",
+    "    def __init__(self):\n",
+    "        # layouts to help with styling\n",
+    "        self.items_layout = widgets.Layout(width='auto')\n",
+    "\n",
+    "        self.box_layout = widgets.Layout(display='flex',\n",
+    "                                          flex_flow='column',\n",
+    "                                          align_items='stretch',\n",
+    "                                          width='50%',\n",
+    "                                          border='solid 1px gray',\n",
+    "                                          padding='0px 30px 20px 30px')\n",
+    "\n",
+    "        # Create all components\n",
+    "        self.ui_title = widgets.HTML(value=\"<h2>Instructor Grading Configuration</h2>\")\n",
+    "\n",
+    "        self.run_button = widgets.Button(description='Submit', button_style='success', icon='check')\n",
+    "        self.status_output = widgets.Output()\n",
+    "        self.status_output.append_stdout('Waiting...')\n",
+    "\n",
+    "        # Setup click behavior\n",
+    "        self.run_button.on_click(self._setup_environment)\n",
+    "\n",
+    "        # Reset rest of state\n",
+    "        self.reset_state()\n",
+    "\n",
+    "    def reset_state(self, close_all=False):\n",
+    "\n",
+    "        if close_all:\n",
+    "            self.learning_objectives_text.close()\n",
+    "            self.file_upload.close()\n",
+    "            self.file_upload_box.close()\n",
+    "            #self.ui_container.close()\n",
+    "\n",
+    "        self.learning_objectives_text = widgets.Textarea(value='', description='Learning Objectives',\n",
+    "                                                         placeholder='Learning objectives: 1. Understand and implement classes in object-oriented programming',\n",
+    "                                                         layout=self.items_layout,\n",
+    "                                                         style={'description_width': 'initial'})\n",
+    "        self.file_upload = widgets.FileUpload(\n",
+    "            accept='.zip',  # Accepted file extension e.g. '.txt', '.pdf', 'image/*', 'image/*,.pdf'\n",
+    "            multiple=False  # True to accept multiple files upload else False\n",
+    "        )\n",
+    "        self.file_upload_box = widgets.HBox([widgets.Label('Upload User Files:\\t'), self.file_upload])\n",
+    "\n",
+    "\n",
+    "        # Create a VBox container to arrange the widgets vertically\n",
+    "        self.ui_container = widgets.VBox([self.ui_title, self.learning_objectives_text,\n",
+    "                                           self.file_upload_box, self.run_button, self.status_output],\n",
+    "                                          layout=self.box_layout)\n",
+    "\n",
+    "\n",
+    "    def _setup_environment(self, btn):\n",
+    "        grade_settings['learning_objectives'] = self.learning_objectives_text.value\n",
+    "        grade_settings['json_file_path'] = self.file_upload.value\n",
+    "\n",
+    "        if self.file_upload.value:\n",
+    "            try:\n",
+    "                input_file = list(self.file_upload.value.values())[0]\n",
+    "                extracted_zip_dir = list(grade_settings['json_file_path'].keys())[0][:-4]\n",
+    "            except:\n",
+    "                input_file = self.file_upload.value[0]\n",
+    "                extracted_zip_dir = self.file_upload.value[0]['name'][:-4]\n",
+    "\n",
+    "            self.status_output.clear_output()\n",
+    "            self.status_output.append_stdout('Loading zip file...\\n')\n",
+    "\n",
+    "            with zipfile.ZipFile(io.BytesIO(input_file['content']), \"r\") as z:\n",
+    "                z.extractall()\n",
+    "                extracted_files = z.namelist()\n",
+    "\n",
+    "            self.status_output.append_stdout('Extracted files and directories: {0}\\n'.format(', '.join(extracted_files)))\n",
+    "\n",
+    "            # load all json files\n",
+    "            grade_settings['json_files'] = glob.glob(''.join([extracted_zip_dir, '/**/*.json']), recursive=True)\n",
+    "\n",
+    "            #status_output.clear_output()\n",
+    "            self.status_output.append_stdout('Loading successful!\\nLearning Objectives: {0}\\nExtracted JSON files: {1}'.format(grade_settings['learning_objectives'],\n",
+    "                                                                                                        ', '.join(grade_settings['json_files'])))\n",
+    "\n",
+    "        else:\n",
+    "            self.status_output.clear_output()\n",
+    "            self.status_output.append_stdout('Please upload a zip file.')\n",
+    "\n",
+    "        # Clear values so they're not saved\n",
+    "        self.learning_objectives_text.value = ''\n",
+    "        self.reset_state(close_all=True)\n",
+    "        self.run_ui_container()\n",
+    "\n",
+    "        with self.status_output:\n",
+    "            print('Extracted files and directories: {0}\\n'.format(', '.join(extracted_files)))\n",
+    "            print('Loading successful!\\nLearning Objectives: {0}\\nExtracted JSON files: {1}'.format(grade_settings['learning_objectives'],\n",
+    "                                                                                                        ', '.join(grade_settings['json_files'])))\n",
+    "            print('Submitted and Reset all values.')\n",
+    "\n",
+    "\n",
+    "    def run_ui_container(self):\n",
+    "        display(self.ui_container, clear=True)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 16
+    },
+    "id": "4wCQ4Wk8YiG4",
+    "outputId": "5c602e80-a210-4449-fd6c-eb8bf3213407"
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "#This code helps in the case that we have problems with metadata being retained.\n",
+    "#!jupyter nbconvert --ClearOutputPreprocessor.enabled=True --ClearMetadataPreprocessor.enabled=True --ClearMetadataPreprocessor.preserve_cell_metadata_mask \"colab\" --ClearMetadataPreprocessor.preserve_cell_metadata_mask \"kernelspec\" --ClearMetadataPreprocessor.preserve_cell_metadata_mask \"language_info\" --to=notebook --output=instructor_inst_notebook.ipynb instructor_intr_notebook.ipynb"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "gj1K3MjHDlqb"
+   },
+   "source": [
+    "# User Settings and Submission Upload\n",
+    "The following two cells will ask you for your OpenAI API credentials and to upload the json file of the student submission."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 519,
+     "referenced_widgets": [
+      "a84d31fb8f4e4bafb74035158834b404",
+      "b051a90758434644955747bc02d00bab",
+      "252b8009f3734ed2908049ebb40c0247",
+      "6622f76f91f44527a87a7575bbd388d2",
+      "654ab6d155eb457ea5c719a9ac27ad5b",
+      "86cb4f568f454ff8832face502fb0745",
+      "e30fe87f01bc4580a61713b5b72439a2",
+      "d16b25c7e9e948938c9303fbe8ae3dcc",
+      "453b12da4b6540cd9e4e57f73a4d670c",
+      "b74cf92175374028948d4cf529d4d1e6",
+      "f7d75b0a32554a9589c513336fc30095",
+      "7f7164e80a464ba9b99f96c10132db25",
+      "49f80567705147f0b82d45b7f06dd1ba",
+      "5a17f4509d194105b23dd616e45183d5",
+      "81c4dda35a7d4e15821bb4bc0973354e",
+      "df1c46361f714aceb9c046f98fede40c",
+      "60b80d550efa403a825a3cb913c26f53",
+      "d0bd0e3f12594ff1a51365b65a3fcc43",
+      "dfa8d6c7d70b42468cbda035de89404c",
+      "26d13984d45745858d3b890bc7f18a90",
+      "53722998fbe64a7c94829b79e8cd69d6",
+      "1b7ee0de15484cd5aecd6d8ca3b6ee9d",
+      "dde20647d3594d31b66b19659f53a95e",
+      "8610fffd2d2a4ec28f8c874c06073ce7",
+      "54e3918921f44fb4a9020beab951fcdf",
+      "1072a8a142f64dfd96ee528a2e9d1595",
+      "67b4083cd4234f52bb7cca27ab9cddb3",
+      "d0a1ebdf7fc0473f91c39b29ca580934",
+      "abbecdc637694e7cb026e003244e7037",
+      "7f814595d31e4b86992b5bd6bc85ced4",
+      "76548751bb9c4bcb9d4f39788ea7d4af",
+      "dbb88901f5084d49af208b91b52b6073"
+     ]
+    },
+    "id": "oQOeYl9OYiG5",
+    "outputId": "bb5b7dc4-ea7b-41ea-e741-2fb2bf66cccc"
+   },
+   "outputs": [
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "1b7ee0de15484cd5aecd6d8ca3b6ee9d",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "VBox(children=(HTML(value='<h2>Instructor Grading Configuration</h2>'), Textarea(value='', description='Learni…"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "InstructorGradingConfig().run_ui_container()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "W9SqmkpeIgpk"
+   },
+   "source": [
+    "You will need an OpenAI API key in order to access the chat functionality. In the following cell, you'll see a blank box pop up - copy your API key there and press enter."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 32
+    },
+    "id": "MK8R5DmEYiG5",
+    "outputId": "09e11ee6-5a9f-4b61-ff61-ddf82a68c498"
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "··········\n"
+     ]
+    }
+   ],
+   "source": [
+    "# setup open AI api key\n",
+    "openai_api_key = getpass()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "collapsed": true,
+    "id": "0bp158bj_0s6"
+   },
+   "source": [
+    "# Execute Grading\n",
+    "Run this cell set to have the generative AI assist you in grading."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "vyJuQ7RUR8tB"
+   },
+   "source": [
+    "## Installation and Loading"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 16
+    },
+    "id": "tjKxWLA3YiG5",
+    "outputId": "6dc85dff-4baa-44f0-edef-42925e6c271a"
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "%%capture\n",
+    "# install additional packages if needed\n",
+    "! pip install -q langchain openai"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 16
+    },
+    "id": "S3oQiNm_YiG5",
+    "outputId": "95e43744-fadc-45cc-bb02-05737d12fcb2"
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "# import necessary libraries here\n",
+    "from langchain.llms import OpenAI\n",
+    "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.prompts import PromptTemplate\n",
+    "from langchain.document_loaders import TextLoader\n",
+    "from langchain.indexes import VectorstoreIndexCreator\n",
+    "from langchain.text_splitter import CharacterTextSplitter\n",
+    "from langchain.embeddings import OpenAIEmbeddings\n",
+    "from langchain.schema import SystemMessage, HumanMessage, AIMessage\n",
+    "import openai"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 16
+    },
+    "id": "uXfSTQPrYiG5",
+    "outputId": "f85a6c4a-009f-4b30-f74a-04b6bfd85af6"
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "# Helper because lines are printed too long; helps with wrapping visualization\n",
+    "from IPython.display import HTML, display\n",
+    "\n",
+    "def set_css():\n",
+    "  display(HTML('''\n",
+    "  <style>\n",
+    "    pre {\n",
+    "        white-space: pre-wrap;\n",
+    "    }\n",
+    "  </style>\n",
+    "  '''))\n",
+    "get_ipython().events.register('pre_run_cell', set_css)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 16
+    },
+    "id": "sTQFW9TxYiG5",
+    "outputId": "e291e167-e635-4006-965d-29fe1a0db10f"
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "# Set pandas display options\n",
+    "pd.set_option('display.max_columns', None)\n",
+    "pd.set_option('display.max_colwidth', 0)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "DOACT_LSSM58"
+   },
+   "source": [
+    "Setting of API key in environment and other settings"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 16
+    },
+    "id": "OV05xRtDYiG5",
+    "outputId": "0d6339d9-bc32-49e9-955f-99947b510456"
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "#extract info from dictionary\n",
+    "json_file_path = grade_settings['json_file_path']\n",
+    "learning_objectives = grade_settings['learning_objectives']\n",
+    "\n",
+    "#set API key\n",
+    "os.environ[\"OPENAI_API_KEY\"] = openai_api_key\n",
+    "openai.api_key = openai_api_key"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "YreIs-I-tuxx"
+   },
+   "source": [
+    "Initiate the OpenAI model using Langchain."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 16
+    },
+    "id": "ZRn9wbJBYiG5",
+    "outputId": "c09c7a7c-1ca0-4860-b39d-91778f183307"
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "llm = ChatOpenAI(model='gpt-3.5-turbo-16k')\n",
+    "messages = [\n",
+    "    SystemMessage(content=\"You are a helpful assistant.\"),\n",
+    "    HumanMessage(content=\"\")\n",
+    "]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "pIKYtr0UTJNc"
+   },
+   "source": [
+    "## Functions to help with loading json"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "t7O3XPC29Osw"
+   },
+   "source": [
+    "`file_upload_json_to_df` helps when you use the file uploader as the json is directly read in this case. `clean_keys` helps when there are errors on the keys when reading."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 16
+    },
+    "id": "qGxPHexrYiG5",
+    "outputId": "80657bb1-97e8-423a-a2ff-99afe8d22718"
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "# Strip beginning and ending newlines\n",
+    "def clean_keys(loaded_json):\n",
+    "  out_json = [{key.strip():value for key, value in json_dict.items()} for json_dict in loaded_json ]\n",
+    "  return out_json\n",
+    "\n",
+    "# Convert difficult datatypes to newlines\n",
+    "def file_upload_json_to_df(upload_json):\n",
+    "\n",
+    "  #get middle key of json to extract content\n",
+    "  fname = list(upload_json.keys())[0]\n",
+    "\n",
+    "  #load the json; strict allows us to get around encoding issues\n",
+    "  loaded_json = json.loads(upload_json[fname]['content'], strict=False)\n",
+    "\n",
+    "  #clean the keys if needed\n",
+    "  loaded_json = clean_keys(loaded_json)\n",
+    "\n",
+    "  return pd.DataFrame(loaded_json)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "N2yuYFQJYiG6"
+   },
+   "source": [
+    "`create_user_dataframe` filters based on role to create a dataframe for only user responses"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 17
+    },
+    "id": "58hygjTXYiG6",
+    "outputId": "8f3683fb-f3da-45f3-8338-772e7583d4cc"
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "def create_user_dataframe(df):\n",
+    "  df_user = df.query(\"`author` == 'user'\")\n",
+    "\n",
+    "  return df_user"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "MOwaLI97Igpm"
+   },
+   "source": [
+    "`load_json_as_df` helps when you use the file uploader as the json is directly read in this case. It accepts the path to the JSON to load the dataframe based on the json."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 131,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 16
+    },
+    "id": "w0xN9CJeYiG6",
+    "outputId": "9422452f-9a97-4f22-9e49-ea0812c298fd"
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "def load_json_as_df(fpath):\n",
+    "    # check if file is .json\n",
+    "    if not fpath.endswith('.json'):\n",
+    "        return None\n",
+    "\n",
+    "    keys = [\"timestamp\", \"author\", \"message\"]\n",
+    "\n",
+    "    df_out = None\n",
+    "    out_error = None\n",
+    "\n",
+    "    try:\n",
+    "        # Read JSON file\n",
+    "        with open(fpath, \"r\") as f:\n",
+    "            json_data = f.read()\n",
+    "\n",
+    "        # Load JSON data\n",
+    "        data = json.loads(json_data, strict=False)\n",
+    "\n",
+    "        # Quick check to see if we can fix common errors in json\n",
+    "        # 1. JSON responses wrapped in enclosing dictionary\n",
+    "        if isinstance(data, dict):\n",
+    "            if len(data.keys()) == 1:\n",
+    "                data = data[list(data.keys())[0]]\n",
+    "            else:\n",
+    "                data = [data]  # convert to list otherwise\n",
+    "\n",
+    "        # We only operate on lists of dictionaries\n",
+    "        if isinstance(data, list):\n",
+    "            data = clean_keys(data)  # clean keys to make sure there are no unnecessary newlines\n",
+    "\n",
+    "            if all(all(k in d for k in keys) for d in data):\n",
+    "                # Filter only the student messages based on the \"author\" key\n",
+    "                data = [d for d in data if d[\"author\"].lower() == \"user\"]\n",
+    "\n",
+    "                df_out = pd.json_normalize(data)\n",
+    "                if len(df_out) <= 1:\n",
+    "                    out_error = [fpath, \"Warning: JSON keys correct, but something wrong with the overall structure of the JSON when converting to the dataframe. The dataframe only has one row. Skipping.\"]\n",
+    "                    df_out = None\n",
+    "            else:\n",
+    "                out_error = [fpath, \"Error: JSON Keys are incorrect. Found keys: \" + str(list(data[0].keys()))]\n",
+    "        else:\n",
+    "            out_error = [fpath, \"Error: Something is wrong with the structure of the JSON.\"]\n",
+    "\n",
+    "    except Exception as e:\n",
+    "        print(f\"Error processing file {fpath}: {str(e)}\")\n",
+    "        out_error = [fpath, \"Fatal System Error: \" + str(e)]\n",
+    "\n",
+    "    if df_out is not None:\n",
+    "        df_out['filename'] = fpath\n",
+    "\n",
+    "    return df_out, out_error"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "N2yuYFQJYiG6"
+   },
+   "source": [
+    "`create_user_dataframe` filters based on role to create a dataframe for only user responses"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 132,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 16
+    },
+    "id": "58hygjTXYiG6",
+    "outputId": "44b588d6-6b6c-4c9e-b944-62ac29117344"
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "def create_user_dataframe(df):\n",
+    "  df_user = df.query(\"`author` == 'user'\")\n",
+    "\n",
+    "  return df_user"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "KA5moX-1Igpn"
+   },
+   "source": [
+    "The `process_file` and `process_files` functions provide the implementation of prompt templates for instructor grading. It uses the input components to assemble a prompt and then sends this prompt to the llm for evaluation alongside the read dataframes."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 15,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 16
+    },
+    "id": "nFz3UVL3YiG6",
+    "outputId": "01029e5f-b313-4941-d572-6dca5903d4ac"
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "def process_file(df, desc, instr, print_results):\n",
+    "    messages_as_string = '\\n'.join(df['message'].astype(str))\n",
+    "    context = messages_as_string\n",
+    "\n",
+    "    # Assemble prompt\n",
+    "    prompt = desc if desc is not None else \"\"\n",
+    "    prompt = (prompt + instr + \"\\n\") if instr is not None else prompt\n",
+    "    prompt = prompt + \"Here is the chat log: \\n\\n\" + context + \"\\n\"\n",
+    "\n",
+    "    # Get results and optionally print\n",
+    "    messages[1] = HumanMessage(content=prompt)\n",
+    "    result = llm(messages)\n",
+    "\n",
+    "    # Check if 'filename' exists in df\n",
+    "    if 'filename' in df:\n",
+    "        if print_results:\n",
+    "            print(f\"\\n\\nResult for file {df['filename'][0]}: \\n{result.content}\")\n",
+    "    else:\n",
+    "        if print_results:\n",
+    "            print(f\"\\n\\nResult for file: Unknown Filename \\n{result.content}\")\n",
+    "\n",
+    "    return result\n",
+    "\n",
+    "def process_files(json_dfs, output_desc=None, grad_instructions=None, use_defaults = False, print_results=True):\n",
+    "    if use_defaults:\n",
+    "        output_desc = (\"Given the following chat log, create a table with the question number, the question content, answer, \"\n",
+    "                       \"whether or not the student answered correctly on the first try, and the number of attempts it took to get the right answer. \")\n",
+    "        grad_instructions = (\"Then, calculate the quiz grade from the total number of assessment questions. \"\n",
+    "                             \"Importantly, a point should only be granted if an answer was correct on the very first attempt. \"\n",
+    "                             \"If an answer was not correct on the first attempt, even if it was correct in subsequent attempts, no point should be awarded for that question. \")\n",
+    "\n",
+    "    results = [process_file(df, output_desc, grad_instructions, print_results) for df in json_dfs]\n",
+    "\n",
+    "    return results"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 16,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 17
+    },
+    "id": "EhryP8utrR9D",
+    "outputId": "51f4a60d-d6b7-4885-85c4-410c741ed651"
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "def output_log_file(df_list, results_list, log_file='evaluation_log.txt'):\n",
+    "    \"\"\"\n",
+    "    Create a single log file containing evaluation results for all students.\n",
+    "\n",
+    "    Parameters:\n",
+    "        df_list (list of pandas.DataFrame): List of DataFrames.\n",
+    "        results_list (list of ai_model_response): List of evaluation results.\n",
+    "        log_file (str): File name where the evaluation log will be saved. Default is 'evaluation_log.txt'.\n",
+    "\n",
+    "    Returns:\n",
+    "        None\n",
+    "    \"\"\"\n",
+    "    with open(log_file, 'w') as log:\n",
+    "        for df, result in zip(df_list, results_list):\n",
+    "            log.write(f\"File: {df['filename'][0]}\\n\")\n",
+    "            log.write(result.content)\n",
+    "            log.write(\"\\n\\n\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "lXQ45cJ1AztR"
+   },
+   "source": [
+    "`pretty_print` makes dataframes look better when printed by substituting non-HTML with HTML for rendering."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 134,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 16
+    },
+    "id": "0te_RLOOYiG6",
+    "outputId": "1dc53c98-5f68-4902-b377-7bba451395f0"
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "def pretty_print(df):\n",
+    "    return display( HTML( df.to_html().replace(\"\\\\n\",\"<br>\") ) )"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "I3rKk7lJYiG6"
+   },
+   "source": [
+    "`save_as_csv` saves the dataframe as a CSV"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 135,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 16
+    },
+    "id": "DnrH2ldeYiG6",
+    "outputId": "f1f6153b-49db-4145-c188-373685ffdcf4"
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "def save_as_csv(df, file_name):\n",
+    "  df.to_csv(file_name, index=False)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 136,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 16
+    },
+    "id": "Vgo_y8R8bzTE",
+    "outputId": "f5effec2-8620-4c1d-be15-672b8cb3de21"
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "def show_json_loading_errors(err_list):\n",
+    "  if err_list:\n",
+    "    print(\"The following files have the following errors upon loading and will NOT be processed:\", '\\n'.join(err_list))\n",
+    "  else:\n",
+    "    print(\"No errors found in uploaded zip JSON files.\")\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "85h5oTysJkHs"
+   },
+   "source": [
+    "## Final data preparation steps"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 137,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 16
+    },
+    "id": "Upah5_ygZRZx",
+    "outputId": "639a0d70-6a93-4462-f65a-2e24f30643c0"
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "#additional processing setup\n",
+    "json_files = grade_settings['json_files']\n",
+    "load_responses = [load_json_as_df(jf) for jf in json_files]\n",
+    "\n",
+    "#unzip to two separate lists\n",
+    "all_json_dfs, errors_list = zip(*load_responses)\n",
+    "\n",
+    "# Remove failed JSONs\n",
+    "all_json_dfs = [df for df in all_json_dfs if df is not None]\n",
+    "\n",
+    "# Update errors list to be individual strings\n",
+    "errors_list = [' '.join(err) for err in errors_list if err is not None]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "P_H4uIfmAsr0"
+   },
+   "source": [
+    "# AI-Assisted Evaluation\n",
+    "Introduction and Instructions\n",
+    "--------------------------------------------------\n",
+    "The following example illustrates how you can specify important components of the prompts for sending to the llm. The `process_files` function will iterate over all of the submissions in your zip file, create dataframes of results (via instruction by setting `output_setup`), and also perform evaluation based on your instructions (via instruction by setting `grading_instructions`).\n",
+    "\n",
+    "Example functionality is demonstrated below."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 138,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 35
+    },
+    "id": "9zIPjG5lco3Z",
+    "outputId": "cc9531c0-3939-4c9f-dc6c-9d2cfa0ea7d2"
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "No errors found in uploaded zip JSON files.\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Print list of files with the incorrect format\n",
+    "show_json_loading_errors(errors_list)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 139,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 1000
+    },
+    "id": "utPzYUoKYiG9",
+    "outputId": "eb3e1769-eb7a-4c93-96bc-6c998df55ef1"
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "Result for file instructorTest/spencer-smith_jesse.json: \n",
+      "Summary and feedback for student responses:\n",
+      "\n",
+      "Student 1:\n",
+      "The student provided an excellent response to Question 1. They accurately explained the purpose of capitalizing expenses when incorporating them into the estimate of corporate earnings. They highlighted the importance of accurately reflecting the timing of costs and their related benefits, and how capitalizing expenses can impact a company's financial statements. The student also mentioned the matching principle of accounting and its role in ensuring the comparability and fairness of financial statements. Overall, the response is comprehensive and well-written. Well done!\n",
+      "\n",
+      "Student 2:\n",
+      "The student gave a great response to Question 2. They correctly stated that expenses should be capitalized when they provide value beyond the current accounting period. The student also provided examples of capital expenses, such as the purchase price of a delivery truck or the cost of a building renovation. These examples demonstrate a clear understanding of the topic. The response is well-explained and shows a good grasp of the concept. Great job!\n",
+      "\n",
+      "Numeric summary:\n",
+      "Both students provided correct answers to their respective questions, earning them a point each. Therefore, the numeric summary is as follows:\n",
+      "Student 1: 1 point\n",
+      "Student 2: 1 point\n",
+      "\n",
+      "\n",
+      "Result for file instructorTest/bell_charreau.json: \n",
+      "Summary:\n",
+      "- The first student's response inaccurately states that capitalizing expenses is done to make the money 'look good' on the earnings report. The assistant provides a detailed explanation of the correct purpose of capitalizing expenses.\n",
+      "- The second student's response partially identifies that capitalized expenses provide benefits for a longer period, but the assistant provides a more comprehensive explanation of what types of expenses should be capitalized and why they are treated differently from regular expenses.\n",
+      "\n",
+      "Feedback for the first student:\n",
+      "The student accurately identified the purpose of capitalizing expenses, but their explanation was not entirely correct. They incorrectly stated that it is done to make the money 'look good' on the earnings report. The assistant provided a clear and detailed explanation of the correct purpose of capitalizing expenses and how it aligns with accounting principles.\n",
+      "\n",
+      "Feedback for the second student:\n",
+      "The student partially identified the types of expenses that should be capitalized and why they are treated differently from regular expenses. However, their explanation was not comprehensive. The assistant provided a more detailed explanation of what types of expenses should be capitalized and why, as well as the concept of depreciation or amortization for spreading out the costs over time.\n",
+      "\n",
+      "Numeric Summary:\n",
+      "The first student's response was partially correct, so they receive 0.5 points.\n",
+      "The second student's response was also partially correct, so they receive 0.5 points.\n",
+      "The total point count is 1.\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Example\n",
+    "output_setup = (\"For each student response given in the following chat log, please generate a summary and detailed feedback for each students' responses,\"\n",
+    "                  \", including what the student did well, and what was done poorly. \"\n",
+    "                  \"Additionally, please filter feedback alphabetically by the name of the student from the filename.\")\n",
+    "grading_instructions = (\"Then, calculate a numeric summary, summing up the point totals, \"\n",
+    "                  \"in which a point is awarded for answering correctly. \")\n",
+    "\n",
+    "# Assuming `file_paths` is a list of file paths.\n",
+    "processed_submissions = process_files(all_json_dfs, output_setup, grading_instructions, use_defaults=False, print_results=True)\n",
+    "\n",
+    "output_log_file(all_json_dfs, processed_submissions)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "Pc1myGweIgpo"
+   },
+   "source": [
+    "## Instructor-Specified Evaluation\n",
+    "Now, you can use the following code to create your settings. Change `output_setup` and `grading_instructions` as desired, making sure to keep the syntax (beginning and ending parentheses,and quotes at the beginning and end of each line) correct. `output_setup` has been copied from the previous cell, but you should fill in `grading_instructions`.\n",
+    "\n",
+    "### File Processing Options\n",
+    "The `process_files` function has a number of settings.\n",
+    "* The first setting must always be `all_json_dfs`, which contains the tabular representation of the json output.\n",
+    "* The other settings should be set by name, and are:\n",
+    "  * **`output_desc`**: Shown as `output_setup` here, this contains the isntructions about how you want to the tabular representation to be set up. Note that you can also leave this off of the function list (just erase it and the following comma).\n",
+    "  * **`grad_instructions`**: Shown as `grading_instructions` here, use this variable to set grading instructions. Note that you can also leave this off of the function list (erase it and the following comma)\n",
+    "  * **`use_defaults`**: Some default grading and instruction prompts have already been created. If you set `use_defaults=TRUE`, both the grading instructions and the output table description will use the default prompts provided by the program, regardless of whether you have set values for `output_desc` or `grad_instructions`.\n",
+    "  * **`print_results`**: By default, the results will be printed for all students. However, if you don't want to see this output, you can set `print_results=False`.\n",
+    "\n",
+    "Again, make sure to observe the syntax. The defaults used in the program are shown in the above example."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 34,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 16
+    },
+    "id": "GiebKVlbYiG9",
+    "outputId": "d4769d23-393c-4986-9418-5cef6944e6ab"
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "output_setup = (\"For each student response given in the following chat log, please generate a summary and detailed feedback for each students' responses,\"\n",
+    "                  \", including what the student did well, and what was done poorly. \")\n",
+    "\n",
+    "# add your own grading instructions\n",
+    "grading_instructions = (\"INSERT ANY CUSTOM GRADING INSTRUCTIONS HERE\")\n",
+    "\n",
+    "# Assuming `file_paths` is a list of file paths.\n",
+    "processed_submissions = process_files(all_json_dfs, output_setup, grading_instructions, use_defaults=False, print_results=True)\n",
+    "\n",
+    "output_log_file(all_json_dfs, processed_submissions)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "snLA6OZ83CrS"
+   },
+   "source": [
+    "## Grading based on Blooms Taxonomy\n",
+    "Another mechanism of evaluation is through Bloom's Taxonomy, where student responses will be evaluated based on where they fall on Bloom's Taxonomy. The higher the score with Bloom's Taxonomy, the more depth is illustrated by the question."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 140,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 625
+    },
+    "id": "HEPXCJdrYiG-",
+    "outputId": "add813e2-6b7c-4772-dac5-5f756e893b8f"
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "Result for file 0    instructorTest/bell_charreau.json      \n",
+      "0    instructorTest/spencer-smith_jesse.json\n",
+      "Name: filename, dtype: object: \n",
+      "Student 1:\n",
+      "Summary: The student incorrectly states that capitalizing expenses is done to make the money look good on the earnings report.\n",
+      "Feedback: The student's response is not accurate. They misunderstood the purpose of capitalizing expenses. The main purpose is to spread the cost of certain long-term assets over their useful life, not to make the money 'look good' on the earnings report. \n",
+      "Overall Level of Engagement and Knowledge: 1 (Remember)\n",
+      "\n",
+      "Student 2:\n",
+      "Summary: The student partially understands the purpose of capitalizing expenses, but their answer could be more comprehensive.\n",
+      "Feedback: The student correctly notes that capitalized expenses provide benefits for a longer period and are different from regular expenses. However, their answer could be more comprehensive and provide a more thorough explanation of why certain expenses are capitalized and how they are treated differently.\n",
+      "Overall Level of Engagement and Knowledge: 3 (Apply)\n",
+      "\n",
+      "Student 3:\n",
+      "Summary: The student provides a comprehensive and accurate explanation of the purpose of capitalizing expenses.\n",
+      "Feedback: The student's response is excellent. They provide a comprehensive explanation of why capitalizing expenses is crucial for incorporating costs into the estimate of corporate earnings. They correctly highlight the timing of costs and their related benefits, the impact on financial statements, and the alignment with the matching principle of accounting.\n",
+      "Overall Level of Engagement and Knowledge: 6 (Create)\n",
+      "\n",
+      "Student 4:\n",
+      "Summary: The student accurately states that expenses should be capitalized when they provide value beyond the current accounting period and provides relevant examples.\n",
+      "Feedback: The student's response is great. They correctly identify the types of expenses that should be capitalized and provide relevant examples. They demonstrate a clear understanding of the concept and provide a well-explained answer.\n",
+      "Overall Level of Engagement and Knowledge: 5 (Evaluate)\n"
+     ]
+    }
+   ],
+   "source": [
+    "output_setup = (\"For each student response given in the following chat log, please generate a summary and detailed feedback for each students' responses,\"\n",
+    "                  \", including what the student did well, and what was done poorly. \")\n",
+    "grading_instructions = \"\"\"\\nEvaluate the each student's overall level or engagement and knowledge, based on bloom's taxonomy using their responses.\n",
+    "Bloom's taxonomy is rated on a 1-6 point system, with 1 being remember (recall facts and basic concepts), 2 being understand (explain ideas or concepts),\n",
+    "3 being apply (use information in new situations), 4 being analyze (draw connections among ideas), 5 being evaluate (justify a stand or decision),\n",
+    "and 6 being create (produce new or original work). Assign the interaction a score from 1-6, where 1 = remember, 2 = understand, 3 = apply, 4 = analyze,\n",
+    "5 = evaluate, and 6 = create.\"\"\"\n",
+    "\n",
+    "# Assuming `file_paths` is a list of file paths.\n",
+    "processed_submissions = process_files(all_json_dfs, output_setup, grading_instructions, use_defaults=False, print_results=True)\n",
+    "\n",
+    "output_log_file(all_json_dfs, processed_submissions)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "FI5-vnUvXM03"
+   },
+   "source": [
+    "# Returning Results\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "LgoGt82CYiG-"
+   },
+   "source": [
+    "**Extract Student Responses ONLY from CHAT JSON**\n",
+    "\n",
+    "Below are relevant user components of dataframes, including the conversion from the original json, the interaction labeled dataframe, and the output dataframe. Check to make sure they make sense."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 16
+    },
+    "id": "HVq9i_mXYiG-",
+    "outputId": "fb5251e9-a327-4dfa-e294-03f5c58f6d35"
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "def write_responses_to_csv(json_dfs):\n",
+    "    # Concatenate all dataframes in json_dfs into one large dataframe\n",
+    "    df = pd.concat(json_dfs)\n",
+    "\n",
+    "    # Write the dataframe to a CSV\n",
+    "    df.to_csv('all_student_responses.csv', index=False)\n",
+    "\n",
+    "write_responses_to_csv(all_json_dfs)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "1WIGxKmDYiG-"
+   },
+   "source": [
+    "**Saving/Downloading AI-Assisted Student Evaluation from Chat JSON**\n",
+    "\n",
+    "Execute the following cell to have all of your students' data returned in a single CSV file."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 16
+    },
+    "id": "QnWNEeqjYiG-",
+    "outputId": "e98c0c39-8449-45d4-f93a-d9394f6781bf"
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "  <style>\n",
+       "    pre {\n",
+       "        white-space: pre-wrap;\n",
+       "    }\n",
+       "  </style>\n",
+       "  "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "# Start with an empty dataframe\n",
+    "all_results_df = pd.DataFrame()\n",
+    "\n",
+    "for result in processed_submissions:\n",
+    "\n",
+    "    # Append the data from the current file to the master dataframe\n",
+    "    all_results_df = pd.concat([all_results_df, df])\n",
+    "\n",
+    "# Now all_results_df contains data from all the files\n",
+    "\n",
+    "# Write all results to a single CSV\n",
+    "all_results_df.to_csv('all_results.csv', index=False)"
+   ]
+  }
+ ],
+ "metadata": {
+  "colab": {
+   "include_colab_link": true,
+   "provenance": []
+  },
+  "kernelspec": {
+   "display_name": "Python 3",
+   "name": "python3"
+  },
+  "language_info": {
+   "name": "python",
+   "version": "3.10.6"
+  },
+  "widgets": {
+   "application/vnd.jupyter.widget-state+json": {
+    "1072a8a142f64dfd96ee528a2e9d1595": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "LabelModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "LabelModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "LabelView",
+      "description": "",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_abbecdc637694e7cb026e003244e7037",
+      "placeholder": "​",
+      "style": "IPY_MODEL_7f814595d31e4b86992b5bd6bc85ced4",
+      "value": "Upload User Files:\t"
+     }
+    },
+    "1b7ee0de15484cd5aecd6d8ca3b6ee9d": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "VBoxModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "VBoxModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "VBoxView",
+      "box_style": "",
+      "children": [
+       "IPY_MODEL_b051a90758434644955747bc02d00bab",
+       "IPY_MODEL_dde20647d3594d31b66b19659f53a95e",
+       "IPY_MODEL_8610fffd2d2a4ec28f8c874c06073ce7",
+       "IPY_MODEL_654ab6d155eb457ea5c719a9ac27ad5b",
+       "IPY_MODEL_86cb4f568f454ff8832face502fb0745"
+      ],
+      "layout": "IPY_MODEL_e30fe87f01bc4580a61713b5b72439a2"
+     }
+    },
+    "252b8009f3734ed2908049ebb40c0247": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "TextareaModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "TextareaModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "TextareaView",
+      "continuous_update": true,
+      "description": "Learning Objectives",
+      "description_tooltip": null,
+      "disabled": false,
+      "layout": "IPY_MODEL_b74cf92175374028948d4cf529d4d1e6",
+      "placeholder": "Learning objectives: 1. Understand and implement classes in object-oriented programming",
+      "rows": null,
+      "style": "IPY_MODEL_f7d75b0a32554a9589c513336fc30095",
+      "value": ""
+     }
+    },
+    "26d13984d45745858d3b890bc7f18a90": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "ButtonStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "ButtonStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "button_color": null,
+      "font_weight": ""
+     }
+    },
+    "453b12da4b6540cd9e4e57f73a4d670c": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "DescriptionStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "DescriptionStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "description_width": ""
+     }
+    },
+    "49f80567705147f0b82d45b7f06dd1ba": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "FileUploadModel",
+     "state": {
+      "_counter": 1,
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "FileUploadModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "FileUploadView",
+      "accept": ".zip",
+      "button_style": "",
+      "data": [
+       null
+      ],
+      "description": "Upload",
+      "description_tooltip": null,
+      "disabled": false,
+      "error": "",
+      "icon": "upload",
+      "layout": "IPY_MODEL_dfa8d6c7d70b42468cbda035de89404c",
+      "metadata": [
+       {
+        "lastModified": 1689919477171,
+        "name": "instructorTest.zip",
+        "size": 4958,
+        "type": "application/zip"
+       }
+      ],
+      "multiple": false,
+      "style": "IPY_MODEL_26d13984d45745858d3b890bc7f18a90"
+     }
+    },
+    "53722998fbe64a7c94829b79e8cd69d6": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "54e3918921f44fb4a9020beab951fcdf": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "DescriptionStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "DescriptionStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "description_width": "initial"
+     }
+    },
+    "5a17f4509d194105b23dd616e45183d5": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "60b80d550efa403a825a3cb913c26f53": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "654ab6d155eb457ea5c719a9ac27ad5b": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "ButtonModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "ButtonModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "ButtonView",
+      "button_style": "success",
+      "description": "Submit",
+      "disabled": false,
+      "icon": "check",
+      "layout": "IPY_MODEL_81c4dda35a7d4e15821bb4bc0973354e",
+      "style": "IPY_MODEL_df1c46361f714aceb9c046f98fede40c",
+      "tooltip": ""
+     }
+    },
+    "6622f76f91f44527a87a7575bbd388d2": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "HBoxModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HBoxModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HBoxView",
+      "box_style": "",
+      "children": [
+       "IPY_MODEL_7f7164e80a464ba9b99f96c10132db25",
+       "IPY_MODEL_49f80567705147f0b82d45b7f06dd1ba"
+      ],
+      "layout": "IPY_MODEL_5a17f4509d194105b23dd616e45183d5"
+     }
+    },
+    "67b4083cd4234f52bb7cca27ab9cddb3": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "FileUploadModel",
+     "state": {
+      "_counter": 0,
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "FileUploadModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "FileUploadView",
+      "accept": ".zip",
+      "button_style": "",
+      "data": [],
+      "description": "Upload",
+      "description_tooltip": null,
+      "disabled": false,
+      "error": "",
+      "icon": "upload",
+      "layout": "IPY_MODEL_76548751bb9c4bcb9d4f39788ea7d4af",
+      "metadata": [],
+      "multiple": false,
+      "style": "IPY_MODEL_dbb88901f5084d49af208b91b52b6073"
+     }
+    },
+    "76548751bb9c4bcb9d4f39788ea7d4af": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "7f7164e80a464ba9b99f96c10132db25": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "LabelModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "LabelModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "LabelView",
+      "description": "",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_60b80d550efa403a825a3cb913c26f53",
+      "placeholder": "​",
+      "style": "IPY_MODEL_d0bd0e3f12594ff1a51365b65a3fcc43",
+      "value": "Upload User Files:\t"
+     }
+    },
+    "7f814595d31e4b86992b5bd6bc85ced4": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "DescriptionStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "DescriptionStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "description_width": ""
+     }
+    },
+    "81c4dda35a7d4e15821bb4bc0973354e": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "8610fffd2d2a4ec28f8c874c06073ce7": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "HBoxModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HBoxModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HBoxView",
+      "box_style": "",
+      "children": [
+       "IPY_MODEL_1072a8a142f64dfd96ee528a2e9d1595",
+       "IPY_MODEL_67b4083cd4234f52bb7cca27ab9cddb3"
+      ],
+      "layout": "IPY_MODEL_d0a1ebdf7fc0473f91c39b29ca580934"
+     }
+    },
+    "86cb4f568f454ff8832face502fb0745": {
+     "model_module": "@jupyter-widgets/output",
+     "model_module_version": "1.0.0",
+     "model_name": "OutputModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/output",
+      "_model_module_version": "1.0.0",
+      "_model_name": "OutputModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/output",
+      "_view_module_version": "1.0.0",
+      "_view_name": "OutputView",
+      "layout": "IPY_MODEL_53722998fbe64a7c94829b79e8cd69d6",
+      "msg_id": "",
+      "outputs": [
+       {
+        "name": "stdout",
+        "output_type": "stream",
+        "text": [
+         "Extracted files and directories: instructorTest/, __MACOSX/._instructorTest, instructorTest/bell_charreau.json, __MACOSX/instructorTest/._bell_charreau.json, instructorTest/spencer-smith_jesse.json, __MACOSX/instructorTest/._spencer-smith_jesse.json\n",
+         "\n",
+         "Loading successful!\n",
+         "Learning Objectives: \n",
+         "Extracted JSON files: instructorTest/spencer-smith_jesse.json, instructorTest/bell_charreau.json\n",
+         "Submitted and Reset all values.\n"
+        ]
+       }
+      ]
+     }
+    },
+    "a84d31fb8f4e4bafb74035158834b404": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "VBoxModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "VBoxModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "VBoxView",
+      "box_style": "",
+      "children": [
+       "IPY_MODEL_b051a90758434644955747bc02d00bab",
+       "IPY_MODEL_252b8009f3734ed2908049ebb40c0247",
+       "IPY_MODEL_6622f76f91f44527a87a7575bbd388d2",
+       "IPY_MODEL_654ab6d155eb457ea5c719a9ac27ad5b",
+       "IPY_MODEL_86cb4f568f454ff8832face502fb0745"
+      ],
+      "layout": "IPY_MODEL_e30fe87f01bc4580a61713b5b72439a2"
+     }
+    },
+    "abbecdc637694e7cb026e003244e7037": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "b051a90758434644955747bc02d00bab": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "HTMLModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HTMLModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HTMLView",
+      "description": "",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_d16b25c7e9e948938c9303fbe8ae3dcc",
+      "placeholder": "​",
+      "style": "IPY_MODEL_453b12da4b6540cd9e4e57f73a4d670c",
+      "value": "<h2>Instructor Grading Configuration</h2>"
+     }
+    },
+    "b74cf92175374028948d4cf529d4d1e6": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": "auto"
+     }
+    },
+    "d0a1ebdf7fc0473f91c39b29ca580934": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "d0bd0e3f12594ff1a51365b65a3fcc43": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "DescriptionStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "DescriptionStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "description_width": ""
+     }
+    },
+    "d16b25c7e9e948938c9303fbe8ae3dcc": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "dbb88901f5084d49af208b91b52b6073": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "ButtonStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "ButtonStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "button_color": null,
+      "font_weight": ""
+     }
+    },
+    "dde20647d3594d31b66b19659f53a95e": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "TextareaModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "TextareaModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "TextareaView",
+      "continuous_update": true,
+      "description": "Learning Objectives",
+      "description_tooltip": null,
+      "disabled": false,
+      "layout": "IPY_MODEL_b74cf92175374028948d4cf529d4d1e6",
+      "placeholder": "Learning objectives: 1. Understand and implement classes in object-oriented programming",
+      "rows": null,
+      "style": "IPY_MODEL_54e3918921f44fb4a9020beab951fcdf",
+      "value": ""
+     }
+    },
+    "df1c46361f714aceb9c046f98fede40c": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "ButtonStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "ButtonStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "button_color": null,
+      "font_weight": ""
+     }
+    },
+    "dfa8d6c7d70b42468cbda035de89404c": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "e30fe87f01bc4580a61713b5b72439a2": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": "stretch",
+      "align_self": null,
+      "border": "solid 1px gray",
+      "bottom": null,
+      "display": "flex",
+      "flex": null,
+      "flex_flow": "column",
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": "0px 30px 20px 30px",
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": "50%"
+     }
+    },
+    "f7d75b0a32554a9589c513336fc30095": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "DescriptionStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "DescriptionStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "description_width": "initial"
+     }
+    }
+   }
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}

instructor_intr_notebook_example_training.ipynb

@@ -0,0 +1,1277 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "view-in-github",
+        "colab_type": "text"
+      },
+      "source": [
+        "<a href=\"https://colab.research.google.com/github/vanderbilt-data-science/lo-achievement/blob/adding_grading_levels_to_instructor_nb/instructor_intr_notebook_example_training.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "WMKrKfx8_3fc"
+      },
+      "source": [
+        "# Instructor Grading and Assessment\n",
+        "This notebook executes grading of student submissions based on the examples provided in the [Wiki](https://github.com/vanderbilt-data-science/lo-achievement/wiki/Examples-of-great,-good,-and-poor-answers-to-questions) from Dr. Jesse Blocher. In this iteration, we use the Unstructured File Loader, which cannot proccess .json files (the preferred format). We are working on finding a file loader that allows .json. In this version of the notebook, the model has only been trained on Question 2 from the notebook.\n",
+        "\n",
+        "To train the model, we used 2 out of the three student example from each grade brack and inputted into a .pdf with clearly defined levels. Then, we used the excluded answers to test the accuracy of the model's grading."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "# Load and Install Neccessary Libraries"
+      ],
+      "metadata": {
+        "id": "2UQgQSoMx4My"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "! pip install -q langchain=='0.0.229' openai gradio numpy chromadb tiktoken unstructured pdf2image pydantic==\"1.10.8\" jq"
+      ],
+      "metadata": {
+        "id": "UJi1Oy0CyPHD"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# import necessary libraries here\n",
+        "from getpass import getpass\n",
+        "from langchain.llms import OpenAI as openai\n",
+        "from langchain.chat_models import ChatOpenAI\n",
+        "from langchain.prompts import PromptTemplate\n",
+        "from langchain.document_loaders import TextLoader\n",
+        "from langchain.indexes import VectorstoreIndexCreator\n",
+        "from langchain.text_splitter import CharacterTextSplitter\n",
+        "from langchain.embeddings import OpenAIEmbeddings\n",
+        "from langchain.schema import SystemMessage, HumanMessage, AIMessage\n",
+        "import numpy as np\n",
+        "import os\n",
+        "from langchain.vectorstores import Chroma\n",
+        "from langchain.document_loaders.unstructured import UnstructuredFileLoader\n",
+        "from langchain.document_loaders import UnstructuredFileLoader\n",
+        "from langchain.chains import VectorDBQA\n",
+        "from langchain.document_loaders import JSONLoader\n",
+        "import json\n",
+        "from pathlib import Path\n",
+        "from pprint import pprint\n",
+        "\n",
+        "\n",
+        "from langchain.prompts.few_shot import FewShotPromptTemplate\n",
+        "from langchain.prompts.prompt import PromptTemplate"
+      ],
+      "metadata": {
+        "id": "YHytCUoExrYe"
+      },
+      "execution_count": 2,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "# Set up model and pass OpenAI Key\n",
+        "Here we are setting up the model and using a system message to pass a persona prompt with the grading advice"
+      ],
+      "metadata": {
+        "id": "4elyN72szz-_"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# setup open AI api key\n",
+        "openai_api_key = getpass()"
+      ],
+      "metadata": {
+        "id": "jVPEFX3ixJnM"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "os.environ[\"OPENAI_API_KEY\"] = openai_api_key\n",
+        "openai.api_key = openai_api_key"
+      ],
+      "metadata": {
+        "id": "obplpeB78h_M"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Initiate model (model type and specify persona)\n",
+        "llm = ChatOpenAI(model='gpt-3.5-turbo-16k')\n",
+        "messages = [\n",
+        "    SystemMessage(content=\"You are a helpful grading assistant. In grading the following questions, keep in mind the advice from the professor: one aspect of it was being specific. The poor answers have a lot of platitudes, the better answers give specific examples. Secondly, they should discuss automation and/or prediction specifically. Those are the things that ML does, it is not 'technology' broadly.\"),\n",
+        "    HumanMessage(content=\"\")\n",
+        "]"
+      ],
+      "metadata": {
+        "id": "e_ZavOnS8iuE"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "# Original route: Vector Stores from .json files\n",
+        "This section uses .json files and vector stores with system/human messaging and a persona prompt tailored to Dr. Blocher's grading philosophy for the Wiki examples (only includes section 2 at this time)."
+      ],
+      "metadata": {
+        "id": "KHS3E6PydN-2"
+      }
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "## Grading based on A, B, and C-level answers from previous students to Question 2 from the [Wiki](https://github.com/vanderbilt-data-science/lo-achievement/wiki/Examples-of-great,-good,-and-poor-answers-to-questions):\n",
+        "\n",
+        "**Question 2:** Why is machine learning so important for businesses? Answer this question generally (i.e. such that it applies to many or at least most businesses)."
+      ],
+      "metadata": {
+        "id": "IYCBSD_8l7uu"
+      }
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "### Creating .json file from case examples (Question 2)\n",
+        "The purpose of this cell is to create a json file based on the previously submitted, graded work of students based on the case file provided by Dr. Blocher in the Wiki. So, here you could create your own file, or for quick demo purposes you can use the zip file in the next section heading."
+      ],
+      "metadata": {
+        "id": "TYlGEusr64kA"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "q2 = 'Question 2: Why is machine learning so important for businesses? Answer this question generally (i.e. such that it applies to many or at least most businesses).'"
+      ],
+      "metadata": {
+        "id": "DAIaKTRYxOlh"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# A-level answers\n",
+        "\n",
+        "q2_A_answer_1 = 'Machine learning is extremely important tool for businesses. It can be used in a variety of ways, but most importantly, it can be used to identify patterns within their data that might not otherwise be identified by human beings. For example, it can understand customer behaviors, optimize logistics, and expand efficiencies throughout the business. Machine learning does not get tired, meaning it can work as long as you want it to. It can sift through massive amounts of data, that no human being can look through in an efficient manner. Machine learning can be used as a tool to identify anomalies when something needs to be checked to save or gain money. The predictions that companies gain from machine learning are cheap, accurate, and automate. These machine learning algorithms can be brought to larger scales to encompass the whole business and its operations. It is important to note, Machine learning is just predictions. Predictions to understand important patterns that could make or break a company since they understand the patterns of their business more. It is an amazing tool, but should be used wisely and carefully because if not, it can expensive, useless, and straight up wrong.'\n",
+        "q2_A_answer_2 = 'Machine learning is important for most of the sectors in business. Overall, it gives the company of an overview about what would be the trend for their business industry, and analyze the customer behavior to help business segment their customers groups. Today, many companies have a vast amount of information generated by behavior, computer, events, people, and devices. This massive amount of data is difficult for human to handle, and even if human manages it, it is not profitable as human labor is expensive. Thanks to machine learning, companies can utilize their in-house or even third-party data to make something useful for their business. In medical analysis, for example, with human, it takes a very long time to find patterns in thousands of MRI scans. On the other hand, machines can detect patterns in seconds by entering data as long as the information is correctly labeled or trained properly. Another example would be segmenting customer group. In marketing department, the business could use unsupervised machine learning to cluster their customer segments to generate personalized contents that are relevant for each of individuals.'\n",
+        "\n",
+        "# List creation\n",
+        "\n",
+        "q2_A_answers_list = [q2_A_answer_1, q2_A_answer_2]\n"
+      ],
+      "metadata": {
+        "id": "yQT6aExSr1dP"
+      },
+      "execution_count": 9,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# B-level answers\n",
+        "\n",
+        "q2_B_answer_1 = 'Companies use ML models to improve different aspects of their business, like manufacturing, hiring, deployment, advertising, etc. The main goal is to improve productive and increase profitability of the company. The ML models are fed with company and externally available data to help the company optimize its departments and in turn become more financially successful/ productive. For example, using purchasing history, the company can predict who to advertise products to, to increase sales.'\n",
+        "q2_B_answer_2 = 'Machine learning allows business to have automated decision, scale, predictive analysis and performance. Machine learning also helps a business have a data strategy. This is how a firm uses data, data infrastructure, governance, etc. to accomplish its strategic goals and maintain/grow their competitive advantage within their industry.'\n",
+        "q2_B_answer_3 = 'The short answer is ML can help make decisions for businesses. To be clarified, ML does not make decisions for businesses. I mean it can, but people have not trusted ML enough yet and ML has not been that good to let it directly make business decisions. Business people only use ML to help themselves get intuitions of how decisions should be made and make predictions of results they might get based on their decisions. For example, if a business tries to launch a new product, it will use ML to test whether it will work or not on a small scale before it is introduced to a large scale. People called this step piloting. In this step, people collect data that is generated by using the pilot product and analyze their next move. They could tweak some features based on the feedback. If they think the data in their interest shows the product performs well, they might predict that this product will be successful when it is introduced on a large scale. Then, they will launch it.'\n",
+        "# List creation\n",
+        "\n",
+        "q2_B_answers_list = [q2_B_answer_1, q2_B_answer_2, q2_B_answer_3]"
+      ],
+      "metadata": {
+        "id": "KB1CmeRwtRvf"
+      },
+      "execution_count": 10,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# C-level answers\n",
+        "\n",
+        "q2_C_answer_1 = 'Machine learning powers many of the services we use today, such as the recommendation systems of Spotify and Netflix; search engines such as Google and Bing; social media such as TikTok and Instagram; voices such as Siri and Alexa, the list can go on. All these examples show that machine learning is already starting to play a pivotal role in today\"s data-rich world. Machines can help us sift through useful information that can lead to big breakthroughs, and we have seen the widespread use of this technology in various industries such as finance, healthcare, insurance, manufacturing, transformational change, etc.'\n",
+        "q2_C_answer_2 = 'As technology advanced, there are tons of new data generated and stored. All industries experienced this surge in data, including business. There is a huge amount of business data stored and waited in the database of each firm and they need solutions to utilize these data. Machine learning is a very promising approach for firms to puts these data in and output a meaning pattern or result that could help the firms with their existing work. This could turn into a working product or provide insights that could enhance the efficiency of the company’s workflow. With machine learning, a firm could either enter a new market with the new product or save time and effort with the meaningful insights. Achieving these possibilities with the data they already owned is a low effort but high reward action. This is the reason machine learning is valued by many businesses recently.'\n",
+        "\n",
+        "# List creation\n",
+        "\n",
+        "q2_C_answers_list = [q2_C_answer_1, q2_C_answer_2]"
+      ],
+      "metadata": {
+        "id": "3diAz43othjc"
+      },
+      "execution_count": 11,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "q2_Q_and_A = [\"Question:\", q2, \"A-level Answers\", q2_A_answers_list, \"B-level Answers\", q2_B_answers_list, \"C-level Answers\", q2_C_answers_list]"
+      ],
+      "metadata": {
+        "id": "oAnrMSU6u9do"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "import json\n",
+        "from google.colab import files\n",
+        "\n",
+        "def save_example_answers(examples, filename='wiki_ABC_Q2examples.json'):\n",
+        "  with open(filename, 'w') as file:\n",
+        "    json.dump(examples, file)\n",
+        "  files.download(filename)\n",
+        "\n",
+        "save_example_answers(q2_Q_and_A)"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/",
+          "height": 17
+        },
+        "id": "B16iYMEnri9s",
+        "outputId": "3e565b3d-804c-4b5e-acc8-efeb955c6c14"
+      },
+      "execution_count": null,
+      "outputs": [
+        {
+          "output_type": "display_data",
+          "data": {
+            "text/plain": [
+              "<IPython.core.display.Javascript object>"
+            ],
+            "application/javascript": [
+              "\n",
+              "    async function download(id, filename, size) {\n",
+              "      if (!google.colab.kernel.accessAllowed) {\n",
+              "        return;\n",
+              "      }\n",
+              "      const div = document.createElement('div');\n",
+              "      const label = document.createElement('label');\n",
+              "      label.textContent = `Downloading \"${filename}\": `;\n",
+              "      div.appendChild(label);\n",
+              "      const progress = document.createElement('progress');\n",
+              "      progress.max = size;\n",
+              "      div.appendChild(progress);\n",
+              "      document.body.appendChild(div);\n",
+              "\n",
+              "      const buffers = [];\n",
+              "      let downloaded = 0;\n",
+              "\n",
+              "      const channel = await google.colab.kernel.comms.open(id);\n",
+              "      // Send a message to notify the kernel that we're ready.\n",
+              "      channel.send({})\n",
+              "\n",
+              "      for await (const message of channel.messages) {\n",
+              "        // Send a message to notify the kernel that we're ready.\n",
+              "        channel.send({})\n",
+              "        if (message.buffers) {\n",
+              "          for (const buffer of message.buffers) {\n",
+              "            buffers.push(buffer);\n",
+              "            downloaded += buffer.byteLength;\n",
+              "            progress.value = downloaded;\n",
+              "          }\n",
+              "        }\n",
+              "      }\n",
+              "      const blob = new Blob(buffers, {type: 'application/binary'});\n",
+              "      const a = document.createElement('a');\n",
+              "      a.href = window.URL.createObjectURL(blob);\n",
+              "      a.download = filename;\n",
+              "      div.appendChild(a);\n",
+              "      a.click();\n",
+              "      div.remove();\n",
+              "    }\n",
+              "  "
+            ]
+          },
+          "metadata": {}
+        },
+        {
+          "output_type": "display_data",
+          "data": {
+            "text/plain": [
+              "<IPython.core.display.Javascript object>"
+            ],
+            "application/javascript": [
+              "download(\"download_8ec6f24c-9653-4335-8c1b-766926434399\", \"wiki_ABC_Q2examples.json\", 5931)"
+            ]
+          },
+          "metadata": {}
+        }
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "### Creating a Vector Store\n",
+        "Here we create a vector store based on the .json file (which you can find at this [link](https://drive.google.com/file/d/1nk6JhbqoUHFie-Ewb436pdV-onlObZUt/view?usp=sharing), but will need to unzip)."
+      ],
+      "metadata": {
+        "id": "exXR-A2oxWeg"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Upload .json or zip\n",
+        "from google.colab import files\n",
+        "uploaded = files.upload()"
+      ],
+      "metadata": {
+        "id": "Wpt6qsmEw8WP"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Unzip file if neccessary\n",
+        "!unzip '/content/wiki_ABC_Q2examples (2).json.zip'"
+      ],
+      "metadata": {
+        "id": "7T2LpkiZh9LT"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Create file path\n",
+        "data = '/content/wiki_ABC_Q2examples (2).json'"
+      ],
+      "metadata": {
+        "id": "CVY0CVvhxyCu"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Load the .json\n",
+        "data = json.loads(Path(file_path).read_text())\n",
+        "data = str(data)"
+      ],
+      "metadata": {
+        "id": "dHjK0nN6yYPH"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Create Vector Store\n",
+        "\n",
+        "text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)\n",
+        "texts = text_splitter.split_text(data)\n",
+        "\n",
+        "embeddings = OpenAIEmbeddings()\n",
+        "\n",
+        "db = Chroma.from_texts(texts, embeddings)\n",
+        "\n",
+        "qa = VectorDBQA.from_chain_type(llm=llm, chain_type=\"stuff\", vectorstore=db, k=1)"
+      ],
+      "metadata": {
+        "id": "paibxyeuxnu1"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Creating examples from student answers to test the Wiki\n",
+        "\n",
+        "example1 = 'The most powerful aspect of machine learning is its ability to automate processes. If the goal is well defined and repeatable, an algorithm can be trained to perform that task far faster than any human and often with more reliability. Because of this, businesses can implement machine learning algorithms to solve problems human labor was previously required for, whether that be mental or physical labor. Although implementing machine learning often has high initial costs, because computers do not require payment outside of maintaining their operation, in the long run companies can save money by either making their workers’ tasks more efficient or by entirely automating tasks. This increases the profit margins for those firms.'\n",
+        "# B level\n",
+        "example2 ='ML systems can help improve access to data while managing compliance. It helps businesspeople deal with data in a more efficient way, since there is numerous data generated per second in such industry. If they use some of the simple classification models with high accuracy, it will save them a lot of time on data cleaning and validation, which are kind of repetitive and time-consuming. For example, NLP provides an easy way for personnel to query business information, understand business processes, and discover new relationships between business data, ideas based on intuition and insight often emerge. Using models to do prediction helps people making wiser decision. Since models can handle way more data than human, newly collected data can feed in the model and get some predictive result as a reference to the decision makers. This is significant because in this industry, time is precious, and traders must decide quickly and precisely. A little negligence will lead to a big mistake, lose a lot of money, and even affect the company\"s reputation. Models can see patterns that are not easy for human to spot, which is also valuable for modify the way people doing analysis and interpret.'\n",
+        "# C level\n",
+        "example3 = 'The machine learning model (or one a broader view, artificial intelligence) is about prediction. According to the lecture, there are tree main innovations in it. Prediction is cheap, more accurate and automated. As a result, armed with machine learning, businesses could automatically gain much more accurate and powerful capabilities in forecasting, leading to a big savings both in time and money.'\n",
+        "\n",
+        "# Randomized list of answers\n",
+        "training_answers = [example2, example1, example3]"
+      ],
+      "metadata": {
+        "id": "-fyY-ftV-ZbG"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "## Grading based on A, B, and C-level answers from previous students to Question 4 from the [Wiki](https://github.com/vanderbilt-data-science/lo-achievement/wiki/Examples-of-great,-good,-and-poor-answers-to-questions):\n",
+        "\n",
+        "**Question 4:** Describe an example where machine learning is not likely to work well. This should be an example where intelligent, rational people could think of using machine learning or even are already implementing it. Describe specifically why you think ML is unlikely to work well. This should not be an example where someone is implementing it badly (i.e. skipping testing). Rather, it should be an example of how ML simply will not work well, no matter how many smart people work carefully on the problem. You may use examples we’ve discussed in class; it does not need to be a novel application."
+      ],
+      "metadata": {
+        "id": "sSnxqnGDmPzC"
+      }
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "### Creating .json file from case examples (Question 4)\n",
+        "The purpose of this cell is to create a json file based on the previously submitted, graded work of students based on the case file provided by Dr. Blocher in the Wiki."
+      ],
+      "metadata": {
+        "id": "RMkTwaEFmhWd"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Context: question\n",
+        "\n",
+        "q4 = 'Question 4: Describe an example where machine learning is not likely to work well. This should be an example where intelligent, rational people could think of using machine learning or even are already implementing it. Describe specifically why you think ML is unlikely to work well. This should not be an example where someone is implementing it badly (i.e. skipping testing). Rather, it should be an example of how ML simply will not work well, no matter how many smart people work carefully on the problem. You may use examples we’ve discussed in class; it does not need to be a novel application.'"
+      ],
+      "metadata": {
+        "id": "_UVEgvzKmltw"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# A-level answers\n",
+        "\n",
+        "q4_A_answer_1 = 'This is one of the cases discussed in class: Machine learning on the management of at-risk children. Traditionally, at-risk children are mostly determined by social workers and they would report to their agencies if some children are at-risk. The at-risk children will subsequentially be taken into custody before their circumstance is evaluated and a decision about their future is reached. If a machine-learning algorithm were to replace the aforementioned social worker: the biggest concern would be when the algorithm outputs a false positive or false negative prediction. A false positive means that the children are said to be at risk while actually not and removed from their households. A false negative would be when some children are not removed from the dangerous household because the algorithm returned negative results. The costs of both incorrect decisions are too big for anyone to afford. Admittedly, data scientists might be able to develop a great algorithm with very low false positive and false negative rates, and that social workers can easily make the same amount of mistakes or more, but it is still too much for our humane society to take by putting such a life-changing or even life-or-death decision in a machine’s metaphorical hands.'\n",
+        "q4_A_answer_2 = 'I believe machine learning will not work well in the court system when making sentences and decisions. I have heard in the news that machine learning was helping judges sentence defenders. Currently, I am unaware if this is still happening but this will not work because our court system has a racial basis. This racial bias stems back hundreds of years and a machine-learning model would exploit the discrimination already so present in our system. The dataset used to train the model has bias that would unfairly sentence a person because of their demographic. This overall is bad because there is no dataset you can use to train a model that already does not include some sort of inherited bias.'\n",
+        "\n",
+        "# List creation\n",
+        "\n",
+        "q4_A_answers_list = [q4_A_answer_1, q4_A_answer_2]\n"
+      ],
+      "metadata": {
+        "id": "Irw0U9PBmupJ"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# B-level answers\n",
+        "\n",
+        "q4_B_answer_1 = 'I believe that machine learning won’t work well in the advent of self-driving cars. As discussed in class, machines are bad at “analogies, small data or rare events, and identifying causality” (Lecture 4: Data Strategy.) While most driving occurrences are habitual and can be predicted, especially day-to-day events like driving to work or the grocery store, so much driving is small or rare events that can’t necessarily be predicted. For example, you can’t predict that the person behind you is texting while driving and is going to rear end you, or that a snowstorm can make you start to hydroplane on the road. So much of the process of learning is understanding the exceptions, such as driving in the rain, at night, or almost avoiding an accident. For self-driving cars to be successful, so much data is required to reduce error, which I don’t think is possible to acquire. While most algorithms can have leeway in their accuracy, self-driving cars cannot have any errors — a difficult feat.'\n",
+        "q4_B_answer_2 = 'As we discussed in class, in a hospital setting, ML cannot do the task of diagnosing medical conditions well. While ML have been used in some diagnostic tasks, such as detecting some certain types of cancer, it generally to be considered difficult and will not be used in a large scale because medical conditions can have many different conditions can have many different symptoms and can be difficult to diagnose even for experienced doctors. The training of a machine learning model for diagnosing medical conditions requires a large dataset of medical records, along with labels indicating each diagnosis. But there are several barriers. First, medical records are often private and sensitive, and sharing them may raise legal and ethical concerns. Second, medical conditions are complicated, it’s hard to ensure the labels are correct, some of which may takes years or decades to testify. Even if they can recognize some pattern for some disease, symptoms vary from person to person. At a first glance, even after running some tests, it’s hard for doctors to tell the case. It will be harder to use ML in diagnose in general.'\n",
+        "\n",
+        "q4_B_answers_list = [q4_B_answer_1, q4_B_answer_2]"
+      ],
+      "metadata": {
+        "id": "haKh_FsenmT3"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# C-level answers\n",
+        "\n",
+        "q4_C_answer_1 = 'One example would be unstable models. Some of the models frequently exhibit extreme instability and degrade over time. In such circumstances, the company can call for model monitoring and high-frequency model revision. Businesses may begin to revert to intuition-based approach when model creation lead times increase.'\n",
+        "q4_C_answer_2 = 'The example I want to describe is about GPS. We have heard that some drivers who blindly followed their GPS into disaster. The news reported that Three tourists from Bellevue, Washington, got lost after midnight in 2011 when they were unable to find their way back to the hotel. They took what they believed to be a road that would take them to the freeway after requesting the GPS to reroute them. Instead, their SUV submerged into a large body of water. These people trust algorithms and data more than their own logic and judgement. People who drive into the water just because they believe too much in GPS. Algorithms will never think like a human being and cannot make moral judgement like a human being. Machine learning is unable to provide any guidance on the accepted norms. Machine minds & human minds could never be the same. Everyone’s answer is all different, We cannot collect everyone’s information around the world, that is so hard. As long as there is one person’s information not counted, then it will have difference from what you predict. The world always require human’s judgement and ethics, that is the point can not be changed.'\n",
+        "q4_C_answer_3 = 'The organization and the abundance of your data is paramount in the applying machine learning. When your data are either incomplete, like not enough data, or unstructured, like messy data with a lot of missingness and inconsistencies, you are not likely to obtain a good result even from implementing the best model in the world. There are some techniques that you might be able to amplify your data or transfer other data into your own, such as transfer learning or domain adaptation. However, it is still strongly recommended that you should have plenty of data to perform machine learning on. Your data should also be able to well represent the question you are trying to answer. Low coverage of the true target population is likely to result in skewness of the result. Machines learning is not likely to work well in this scenario.'\n",
+        "\n",
+        "# List creation\n",
+        "\n",
+        "q4_C_answers_list = [q4_C_answer_1, q4_C_answer_2, q4_C_answer_3]"
+      ],
+      "metadata": {
+        "id": "YNHULek7oCE_"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "q4_Q_and_A = [\"Question:\", q4, \"A-level Answers\", q4_A_answers_list, \"B-level Answers\", q4_B_answers_list, \"C-level Answers\", q4_C_answers_list]"
+      ],
+      "metadata": {
+        "id": "9bXSVmPnojIh"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "import json\n",
+        "from google.colab import files\n",
+        "\n",
+        "def save_example_answers(examples, filename='wiki_ABC_Q4examples.json'):\n",
+        "  with open(filename, 'w') as file:\n",
+        "    json.dump(examples, file)\n",
+        "  files.download(filename)\n",
+        "\n",
+        "save_example_answers(q4_Q_and_A)"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/",
+          "height": 17
+        },
+        "outputId": "3e565b3d-804c-4b5e-acc8-efeb955c6c14",
+        "id": "Qqwk0mvroqWz"
+      },
+      "execution_count": null,
+      "outputs": [
+        {
+          "output_type": "display_data",
+          "data": {
+            "text/plain": [
+              "<IPython.core.display.Javascript object>"
+            ],
+            "application/javascript": [
+              "\n",
+              "    async function download(id, filename, size) {\n",
+              "      if (!google.colab.kernel.accessAllowed) {\n",
+              "        return;\n",
+              "      }\n",
+              "      const div = document.createElement('div');\n",
+              "      const label = document.createElement('label');\n",
+              "      label.textContent = `Downloading \"${filename}\": `;\n",
+              "      div.appendChild(label);\n",
+              "      const progress = document.createElement('progress');\n",
+              "      progress.max = size;\n",
+              "      div.appendChild(progress);\n",
+              "      document.body.appendChild(div);\n",
+              "\n",
+              "      const buffers = [];\n",
+              "      let downloaded = 0;\n",
+              "\n",
+              "      const channel = await google.colab.kernel.comms.open(id);\n",
+              "      // Send a message to notify the kernel that we're ready.\n",
+              "      channel.send({})\n",
+              "\n",
+              "      for await (const message of channel.messages) {\n",
+              "        // Send a message to notify the kernel that we're ready.\n",
+              "        channel.send({})\n",
+              "        if (message.buffers) {\n",
+              "          for (const buffer of message.buffers) {\n",
+              "            buffers.push(buffer);\n",
+              "            downloaded += buffer.byteLength;\n",
+              "            progress.value = downloaded;\n",
+              "          }\n",
+              "        }\n",
+              "      }\n",
+              "      const blob = new Blob(buffers, {type: 'application/binary'});\n",
+              "      const a = document.createElement('a');\n",
+              "      a.href = window.URL.createObjectURL(blob);\n",
+              "      a.download = filename;\n",
+              "      div.appendChild(a);\n",
+              "      a.click();\n",
+              "      div.remove();\n",
+              "    }\n",
+              "  "
+            ]
+          },
+          "metadata": {}
+        },
+        {
+          "output_type": "display_data",
+          "data": {
+            "text/plain": [
+              "<IPython.core.display.Javascript object>"
+            ],
+            "application/javascript": [
+              "download(\"download_8ec6f24c-9653-4335-8c1b-766926434399\", \"wiki_ABC_Q2examples.json\", 5931)"
+            ]
+          },
+          "metadata": {}
+        }
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "## Grading based on A, B, and C-level answers from previous students to Question 5 from the [Wiki](https://github.com/vanderbilt-data-science/lo-achievement/wiki/Examples-of-great,-good,-and-poor-answers-to-questions):\n",
+        "\n",
+        "**Question 5:** Describe a new example where you think machine learning could work. This cannot be examples we’ve given in class like customer churn, fraud detection, image or object recognition, disease diagnosis, stock prediction, self-driving cars, etc. t does not need to be completely novel, e.g., you can look for news stories and find something new. I want something that is different from the commonly used examples. You may use an example that you or another student brought up in class if it meets these criteria. Be sure to explain why you think it is an example where ML could work. Note: You do not need to be proven right that ML works. I am mostly interested in your logic and arguments of why you think it is a good idea, drawing on what we’ve discussed in class."
+      ],
+      "metadata": {
+        "id": "RfofiUreov7C"
+      }
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "### Creating .json file from case examples (Question 5)\n",
+        "The purpose of this cell is to create a json file based on the previously submitted, graded work of students based on the case file provided by Dr. Blocher in the Wiki."
+      ],
+      "metadata": {
+        "id": "P2h8XtXQo6eO"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Context: question\n",
+        "\n",
+        "q5 = 'Question 5: Describe a new example where you think machine learning could work. This cannot be examples we’ve given in class like customer churn, fraud detection, image or object recognition, disease diagnosis, stock prediction, self-driving cars, etc. t does not need to be completely novel, e.g., you can look for news stories and find something new. I want something that is different from the commonly used examples. You may use an example that you or another student brought up in class if it meets these criteria. Be sure to explain why you think it is an example where ML could work. Note: You do not need to be proven right that ML works. I am mostly interested in your logic and arguments of why you think it is a good idea, drawing on what we’ve discussed in class.'"
+      ],
+      "metadata": {
+        "id": "evpoXMUSpCi4"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# A-level answers\n",
+        "\n",
+        "q5_A_answer_1 = 'As I recall, our group discussed that using ML to do intrusion detection for computers. Nowadays, people’s information is digitalized and stored in databases. Since databases are also servers (i.e., computers), there must be some vulnerability. Software is more common to do detection for now, but it can just recognize the intrusion pattern that already known or preset by the programmer. However, this can be accomplished by employing algorithms that can analyze large amounts of data and detect anomalies that may indicate an attempted intrusion. We can feed ML model with network traffic, labeled or not labeled, system logs to let it learn and identify unusual patterns or behavior. With well-learned models, we can use them to find out signs of unauthorized access or even malware. One of the greatest things is that ML model may be able to find some hidden pattern that human has not and predict how and when the intrusion will take place, or even figure out how the malware will function to attach. This trait is one of the most important traits that distinguish ML model from regular software.'\n",
+        "q5_A_answer_2 = 'Machine learning can be used in NLP (natural language processing). It is very useful to deal with massive unstructured text data. For example, we want to build a machine learning model to reflect the personality of Einstein according to the letters he wrote in the past decades to his family and friends. We can use machine learning models for NLP to transfer the unstructured text data to usable data first. Then we will use these data to train the model. After training the model by the letters Einstein wrote in the past decades, the model will have the same or similar personality as Einstein. And now we input the test letters into the model again, we expect the model will give us the sentiment scores of each letter (for example: humor-0.732, joyful-0.193, fear-0.015, sad-0.002). After we got the scores for each letter, we use statistical methods to analyze the scores and we will finally get an aggregate personality of Einstein.'\n",
+        "q5_A_answer_3 = 'The sports world is a place where I generally think machine learning can be well-utilized, but not in all cases. Something I’m thinking about is fantasy football, in which football fans can draft a team and decide who to “start” on their team for a given week and who to “sit.” NFL pundits and commentators can absolutely give ideas on who would be optimal, and do a great job at it, but when it comes to actual predictions for how many points a particular player might score in a given week, based on opponent average stats, expected workload, and other factors, while also measuring that against every other potential player, having an algorithm to back this up is extremely helpful. Sports betting can also benefit from these types of algorithms, and being able to accurately predict something as volatile as sports could have major business (and fandom) implications.'\n",
+        "\n",
+        "# List creation\n",
+        "\n",
+        "q5_A_answers_list = [q5_A_answer_1, q5_A_answer_2, q5_A_answer_3]\n"
+      ],
+      "metadata": {
+        "id": "B9abYhEupKA_"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# B-level answers\n",
+        "\n",
+        "q5_B_answer_1 = 'Machine learning can work well when it comes to finance organization, planning, and tracking. Since machine learning works well with statistical inference, it’s useful in decision making. For things such as planning and organizing, one example that machine learning can work well in is identifying price values for toiletries, groceries, and necessity items across various store within a specific location and output the best location for the user. We know that certain retailers mark up prices for the items on their shelfs. Some items are more expensive in one location than others and often time, certain prices are not posted online. In addition to that, traveling to different stores and compare prices are very inefficient because it can waste gas and time. To solve the problem, the algorithm will examine the user’s location and the item(s) that the user wants and output the best stores for the items. How this item work is that they take in data from all the closest stores, rank each item based on value, distance, and availability. On top of that, if users compile a lists of necessity items and output them in the model, the model can use data and compare stores that have more than one item on the list plus the other features (value, distance, and availability), and recommend the place, or places if item are not available, that could cut down time and expense for the users.'\n",
+        "q5_B_answer_2 = 'Natural gas consumption has rebounded in the last year, with economic recovery and increased extreme weather leading to increased demand for natural gas, resulting in a tight market and soaring prices. Gas supply agencies want to use ML to predict the deliverability of natural gas storage in depleted reservoirs. First, natural gas is most commonly stored in underground formations: 1) depleted reservoirs, 2) aquifers, and 3) salt caverns. A depleted reservoir must have some elements of a good oil and gas formation to be converted to a subsurface natural gas reservoir, such as good porosity and permeability, the presence of good seal rock, and the presence of cap rock. I think this thing can be achieved with ML because there is a huge amount of data collected from natural gas extraction sites around the world since 1915, and this data provides a good basis and realizability for our research. We can test and build models from this data so that we can use ML to make predictions.'\n",
+        "\n",
+        "# List creation\n",
+        "\n",
+        "q5_B_answers_list = [q5_B_answer_1, q5_B_answer_2]"
+      ],
+      "metadata": {
+        "id": "r0_Tt0lspkQ-"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# C-level answers\n",
+        "\n",
+        "q5_C_answer_1 = 'One example where ML would work in is speech recognition, converting speech into text.'\n",
+        "q5_C_answer_2 = 'ML may work well with education industry. Teachers can utilize ML for lesson preparation, such as analyze academic data, recommendation of high-quality teaching resources, on-demand generation of teaching plans. Others might be like intelligent teaching assistants instead of an actual person TA, or dynamically adjust the teaching content based on comprehensive factors such as learning situation analysis and learner style, and student feedback.'\n",
+        "\n",
+        "# List creation\n",
+        "\n",
+        "q5_C_answers_list = [q5_C_answer_1, q5_C_answer_2]"
+      ],
+      "metadata": {
+        "id": "oRlId1mtp46L"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "q5_Q_and_A = [\"Question:\", q5, \"A-level Answers\", q5_A_answers_list, \"B-level Answers\", q5_B_answers_list, \"C-level Answers\", q5_C_answers_list]"
+      ],
+      "metadata": {
+        "id": "XJC2UNxSqHBp"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "import json\n",
+        "from google.colab import files\n",
+        "\n",
+        "def save_example_answers(examples, filename='wiki_ABC_Q5examples.json'):\n",
+        "  with open(filename, 'w') as file:\n",
+        "    json.dump(examples, file)\n",
+        "  files.download(filename)\n",
+        "\n",
+        "save_example_answers(q5_Q_and_A)"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/",
+          "height": 17
+        },
+        "outputId": "3e565b3d-804c-4b5e-acc8-efeb955c6c14",
+        "id": "MRcW9_1MqPG-"
+      },
+      "execution_count": null,
+      "outputs": [
+        {
+          "output_type": "display_data",
+          "data": {
+            "text/plain": [
+              "<IPython.core.display.Javascript object>"
+            ],
+            "application/javascript": [
+              "\n",
+              "    async function download(id, filename, size) {\n",
+              "      if (!google.colab.kernel.accessAllowed) {\n",
+              "        return;\n",
+              "      }\n",
+              "      const div = document.createElement('div');\n",
+              "      const label = document.createElement('label');\n",
+              "      label.textContent = `Downloading \"${filename}\": `;\n",
+              "      div.appendChild(label);\n",
+              "      const progress = document.createElement('progress');\n",
+              "      progress.max = size;\n",
+              "      div.appendChild(progress);\n",
+              "      document.body.appendChild(div);\n",
+              "\n",
+              "      const buffers = [];\n",
+              "      let downloaded = 0;\n",
+              "\n",
+              "      const channel = await google.colab.kernel.comms.open(id);\n",
+              "      // Send a message to notify the kernel that we're ready.\n",
+              "      channel.send({})\n",
+              "\n",
+              "      for await (const message of channel.messages) {\n",
+              "        // Send a message to notify the kernel that we're ready.\n",
+              "        channel.send({})\n",
+              "        if (message.buffers) {\n",
+              "          for (const buffer of message.buffers) {\n",
+              "            buffers.push(buffer);\n",
+              "            downloaded += buffer.byteLength;\n",
+              "            progress.value = downloaded;\n",
+              "          }\n",
+              "        }\n",
+              "      }\n",
+              "      const blob = new Blob(buffers, {type: 'application/binary'});\n",
+              "      const a = document.createElement('a');\n",
+              "      a.href = window.URL.createObjectURL(blob);\n",
+              "      a.download = filename;\n",
+              "      div.appendChild(a);\n",
+              "      a.click();\n",
+              "      div.remove();\n",
+              "    }\n",
+              "  "
+            ]
+          },
+          "metadata": {}
+        },
+        {
+          "output_type": "display_data",
+          "data": {
+            "text/plain": [
+              "<IPython.core.display.Javascript object>"
+            ],
+            "application/javascript": [
+              "download(\"download_8ec6f24c-9653-4335-8c1b-766926434399\", \"wiki_ABC_Q2examples.json\", 5931)"
+            ]
+          },
+          "metadata": {}
+        }
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        " # Exploring with Langchain Example Sets, FewShotPrompts and Example Selectors\n",
+        " Here we're exploring another avenue for this goal to see if we can yield better results than the vector stores. Again, we are using question 2 from the Wiki to test."
+      ],
+      "metadata": {
+        "id": "PpKRr5Vw4_0F"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Context: question\n",
+        "\n",
+        "q2 = 'Question 2: Why is machine learning so important for businesses? Answer this question generally (i.e. such that it applies to many or at least most businesses).'"
+      ],
+      "metadata": {
+        "id": "-kmMFUaLs_Q1"
+      },
+      "execution_count": 7,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "from langchain.prompts.few_shot import FewShotPromptTemplate\n",
+        "from langchain.prompts.prompt import PromptTemplate\n",
+        "\n",
+        "examples = [\n",
+        "  {\n",
+        "    \"question\": f\"\"\" Please grade the following student's answer to the question ({q2})\n",
+        "     on an A-, B-, C-level grading scale: {q2_A_answer_1}.\"\"\",\n",
+        "    \"answer\": \"This student should recieve an A-level grade.\"\n",
+        "  },\n",
+        "  {\n",
+        "    \"question\": f\"\"\" Please grade the following student's answer to the question ({q2})\n",
+        "     on an A-, B-, C-level grading scale: {q2_A_answer_2}.\"\"\",\n",
+        "    \"answer\": \"The student should recieve an A-level grade.\"\n",
+        "  },\n",
+        "  {\n",
+        "    \"question\": f\"\"\" Please grade the following student's answer to the question ({q2})\n",
+        "     on an A-, B-, C-level grading scale: {q2_B_answer_1}.\"\"\",\n",
+        "    \"answer\": \"This student should recieve a B-level grade.\"\n",
+        "  },\n",
+        "  {\n",
+        "    \"question\": f\"\"\" Please grade the following student's answer to the question ({q2})\n",
+        "     on an A-, B-, C-level grading scale: {q2_B_answer_2}.\"\"\",\n",
+        "    \"answer\": \"This student should recieve a B-level grade.\"\n",
+        "  },\n",
+        "  {\n",
+        "    \"question\": f\"\"\" Please grade the following student's answer to the question ({q2})\n",
+        "     on an A-, B-, C-level grading scale: {q2_B_answer_3}.\"\"\",\n",
+        "    \"answer\": \"This student should recieve a B-level grade.\"\n",
+        "  },\n",
+        "  {\n",
+        "    \"question\": f\"\"\" Please grade the following student's answer to the question ({q2})\n",
+        "     on an A-, B-, C-level grading scale: {q2_C_answer_1}.\"\"\",\n",
+        "    \"answer\": \"This student should recieve a C-level grade.\"\n",
+        "  },\n",
+        "  {\n",
+        "    \"question\": f\"\"\" Please grade the following student's answer to the question ({q2})\n",
+        "     on an A-, B-, C-level grading scale: {q2_C_answer_2}.\"\"\",\n",
+        "    \"answer\": \"This student should recieve a C-level grade.\"\n",
+        "  }\n",
+        "]"
+      ],
+      "metadata": {
+        "id": "1idnQLYW-697"
+      },
+      "execution_count": 12,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Example selector\n",
+        "from langchain.prompts.example_selector import SemanticSimilarityExampleSelector, MaxMarginalRelevanceExampleSelector, NGramOverlapExampleSelector\n",
+        "from langchain.vectorstores import Chroma\n",
+        "from langchain.embeddings import OpenAIEmbeddings\n",
+        "\n",
+        "\n",
+        "example_selector_semantic = SemanticSimilarityExampleSelector.from_examples(\n",
+        "    # This is the list of examples available to select from.\n",
+        "    examples,\n",
+        "    # This is the embedding class used to produce embeddings which are used to measure semantic similarity.\n",
+        "    OpenAIEmbeddings(),\n",
+        "    # This is the VectorStore class that is used to store the embeddings and do a similarity search over.\n",
+        "    Chroma,\n",
+        "    # This is the number of examples to produce.\n",
+        "    k=1\n",
+        ")\n",
+        "\n",
+        "example_selector_mmr = MaxMarginalRelevanceExampleSelector.from_examples(\n",
+        "    # This is the list of examples available to select from.\n",
+        "    examples,\n",
+        "    # This is the embedding class used to produce embeddings which are used to measure semantic similarity.\n",
+        "    OpenAIEmbeddings(),\n",
+        "    # This is the VectorStore class that is used to store the embeddings and do a similarity search over.\n",
+        "    Chroma,\n",
+        "    # This is the number of examples to produce.\n",
+        "    k=1,\n",
+        ")\n",
+        "\n",
+        "example_selector_ngram = NGramOverlapExampleSelector(\n",
+        "    # These are the examples it has available to choose from.\n",
+        "    examples=examples,\n",
+        "    # This is the PromptTemplate being used to format the examples.\n",
+        "    example_prompt=example_prompt,\n",
+        "    # This is the threshold, at which selector stops.\n",
+        "    # It is set to -1.0 by default.\n",
+        "    threshold=-1.0,\n",
+        "    # For negative threshold:\n",
+        "    # Selector sorts examples by ngram overlap score, and excludes none.\n",
+        "    # For threshold greater than 1.0:\n",
+        "    # Selector excludes all examples, and returns an empty list.\n",
+        "    # For threshold equal to 0.0:\n",
+        "    # Selector sorts examples by ngram overlap score,\n",
+        "    # and excludes those with no ngram overlap with input.\n",
+        ")\n",
+        "\n"
+      ],
+      "metadata": {
+        "id": "Y7vNiHKhuIjm"
+      },
+      "execution_count": 15,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "example_prompt = PromptTemplate(input_variables=[\"question\", \"answer\"], template=\"Question: {question}\\n{answer}\")\n",
+        "\n",
+        "print(example_prompt.format(**examples[0]))"
+      ],
+      "metadata": {
+        "id": "x2dCmQdACwGi"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "prompt = FewShotPromptTemplate(\n",
+        "    #examples=examples,\n",
+        "    example_selector = example_selector_semantic,\n",
+        "    example_prompt = example_prompt,\n",
+        "    suffix=\"Question: {input}\",\n",
+        "    input_variables=[\"input\"]\n",
+        ")\n",
+        "\n",
+        "print(prompt.format(input=f\"\"\" Please grade the following student's answer to the question ({q2})\n",
+        "     on an A-, B-, C-level grading scale: {example1}.\"\"\"))"
+      ],
+      "metadata": {
+        "id": "Uc_BvY0rBZgx"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "prompt_semantic = FewShotPromptTemplate(\n",
+        "    #examples=examples,\n",
+        "    example_selector = example_selector_semantic,\n",
+        "    example_prompt = example_prompt,\n",
+        "    suffix=\"Question: {input}\",\n",
+        "    input_variables=[\"input\"]\n",
+        ")\n",
+        "\n",
+        "prompt_mmr = FewShotPromptTemplate(\n",
+        "    #examples=examples,\n",
+        "    example_selector = example_selector_mmr,\n",
+        "    example_prompt = example_prompt,\n",
+        "    suffix=\"Question: {input}\",\n",
+        "    input_variables=[\"input\"]\n",
+        ")\n",
+        "\n",
+        "prompt_ngram = FewShotPromptTemplate(\n",
+        "    #examples=examples,\n",
+        "    example_selector = example_selector_ngram,\n",
+        "    example_prompt = example_prompt,\n",
+        "    suffix=\"Question: {input}\",\n",
+        "    input_variables=[\"input\"]\n",
+        ")"
+      ],
+      "metadata": {
+        "id": "k1laCoAPxhcH"
+      },
+      "execution_count": 18,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# First using the Example Set to test without an example selector, just the base OpenAI model.\n",
+        "from langchain.llms import OpenAI\n",
+        "base_llm = OpenAI()\n",
+        "base_llm(prompt.format(input=f\"\"\" Please grade the following students' answers to the question ({q2})\n",
+        "     on an A-, B-, C-level grading scale: student 1: {example1}, student 2: {example2}, and student 3: {example3}. Grade based on the following advice from the professor:\n",
+        "     One aspect of it was being specific. The poor answers (C) have a lot of platitudes, the better answers give\n",
+        "     specific examples (B and A). Secondly, they should discuss automation and/or prediction specifically. Those are\n",
+        "     the things that ML does (A and B answers), it is not 'technology' broadly (what a C answer might say).\n",
+        "     The difference between A and B answers should be in their complexity, specificity, thoughfulness\n",
+        "     , and general insight on the subject. However, keep in mind that these are short answer questions.\n",
+        "     The number of examples or the legnth of the response is not as important as the content itself. After assigning your grade, provide a brief\n",
+        "     explanation on your reasoning for assigning the grade based on the professor's advice, and\n",
+        "     what differentiated the answers from the higher grade level (if applicable). There should be three grades assigned in your response, and assign based on the order the\n",
+        "     answers were passed in.\"\"\"))"
+      ],
+      "metadata": {
+        "id": "rUdx2axQuec1"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Example Selector 1:\n",
+        "\n",
+        "base_llm(prompt_mmr.format(input=f\"\"\" Please grade the following students' answers to the question ({q2})\n",
+        "     on an A-, B-, C-level grading scale: student 1: {example1}, student 2: {example2}, and student 3: {example3}. Grade based on the following advice from the professor:\n",
+        "     One aspect of it was being specific. The poor answers (C) have a lot of platitudes, the better answers give\n",
+        "     specific examples (B and A). Secondly, they should discuss automation and/or prediction specifically. Those are\n",
+        "     the things that ML does (A and B answers), it is not 'technology' broadly (what a C answer might say).\n",
+        "     The difference between A and B answers should be in their complexity, specificity, thoughfulness\n",
+        "     , and general insight on the subject. However, keep in mind that these are short answer questions.\n",
+        "     The number of examples or the legnth of the response is not as important as the content itself. After assigning your grade, provide a brief\n",
+        "     explanation on your reasoning for assigning the grade based on the professor's advice, and\n",
+        "     what differentiated the answers from the higher grade level (if applicable). There should be three grades assigned in your response, and assign based on the order the\n",
+        "     answers were passed in.\"\"\"))"
+      ],
+      "metadata": {
+        "id": "fe24IOslyITA"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Example selector 2:\n",
+        "\n",
+        "base_llm(prompt_ngram.format(input=f\"\"\" Please grade the following students' answers to the question ({q2})\n",
+        "     on an A-, B-, C-level grading scale: student 1: {example1}, student 2: {example2}, and student 3: {example3}. Grade based on the following advice from the professor:\n",
+        "     One aspect of it was being specific. The poor answers (C) have a lot of platitudes, the better answers give\n",
+        "     specific examples (B and A). Secondly, they should discuss automation and/or prediction specifically. Those are\n",
+        "     the things that ML does (A and B answers), it is not 'technology' broadly (what a C answer might say).\n",
+        "     The difference between A and B answers should be in their complexity, specificity, thoughfulness\n",
+        "     , and general insight on the subject. However, keep in mind that these are short answer questions.\n",
+        "     The number of examples or the legnth of the response is not as important as the content itself. After assigning your grade, provide a brief\n",
+        "     explanation on your reasoning for assigning the grade based on the professor's advice, and\n",
+        "     what differentiated the answers from the higher grade level (if applicable). There should be three grades assigned in your response, and assign based on the order the\n",
+        "     answers were passed in.\"\"\"))"
+      ],
+      "metadata": {
+        "id": "dXwJa30wyL4S"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "### Results from the ngram example selector with the following prompt (yielded the best results (relatively):\n",
+        "\n",
+        "\n",
+        "\n",
+        "```\n",
+        "input=f\"\"\" Please grade the following student's answer to the question ({q2})\n",
+        "     on an A-, B-, C-level grading scale: {example3}. Grade based on the following advice from the professor:\n",
+        "     One aspect of it was being specific. The poor answers have a lot of platitudes, the better answers give\n",
+        "     specific examples. Secondly, they should discuss automation and/or prediction specifically. Those are\n",
+        "     the things that ML does, it is not 'technology' broadly. The best answers\n",
+        "     (A) should be more complex and specific in answering the question and meeting that criteria\n",
+        "     than the better answers (B), though length may not be a determinant. After assigning your grade, provide a brief\n",
+        "     explanation on your reasoning for assigning the grade based on the professor's advice.\"\"\"\n",
+        "```\n",
+        "\n",
+        "\n",
+        "\n",
+        "1. Example 1 (A-level grade): \"This student should recieve a B-level grade. The student's answer includes specific examples of how machine learning can be used in business to increase efficiency and profit margins, as well as how machine learning can automate processes previously done by humans. The student also mentions the importance of prediction in machine learning, which is one of its core functions.\" (incorrect)\n",
+        "2. Example 2 (B-level grade): \"This student should recieve a B-level grade. The student's answer provides specific examples of how ML can be used in business such as NLP and predictive analysis. The student also references automation and prediction which are key aspects of ML.\" (correct)\n",
+        "3. Example 3 (C-level grade): \"This student should recieve a B-level grade. The student's answer provides a specific example about how machine learning can help businesses save time and money by providing more accurate predictions. It also mentions the three main innovations that machine learning brings to the table (prediction is cheap, more accurate, and automated).\" (incorrect)"
+      ],
+      "metadata": {
+        "id": "bKqtzH8hcoHD"
+      }
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "# Exploring Chat Prompt Templates and System/Human Messages\n",
+        "This section is unfinalized as we are not sure if it is neccessary for the goals of this project, and we ran into errors which we were unsuccessful in debugging. This approach may yield better results but would need to be further devloped and debugged."
+      ],
+      "metadata": {
+        "id": "YGWq7cw697Wv"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "from langchain.chat_models import ChatOpenAI\n",
+        "from langchain import PromptTemplate, LLMChain\n",
+        "from langchain.prompts.chat import (\n",
+        "    ChatPromptTemplate,\n",
+        "    SystemMessagePromptTemplate,\n",
+        "    AIMessagePromptTemplate,\n",
+        "    HumanMessagePromptTemplate,\n",
+        ")\n",
+        "from langchain.schema import AIMessage, HumanMessage, SystemMessage\n",
+        "\n",
+        "\n",
+        "\n",
+        "from langchain.prompts import (\n",
+        "    ChatPromptTemplate,\n",
+        "    PromptTemplate,\n",
+        "    SystemMessagePromptTemplate,\n",
+        "    AIMessagePromptTemplate,\n",
+        "    HumanMessagePromptTemplate,\n",
+        ")\n",
+        "from langchain.schema import (\n",
+        "    AIMessage,\n",
+        "    HumanMessage,\n",
+        "    SystemMessage\n",
+        ")"
+      ],
+      "metadata": {
+        "id": "NzK7Yo_OEqlG"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "template=\"You are a helpful assistant that grades student responses ({answer}) to questions ({question}) about machine learning.\"\n",
+        "system_message_prompt = SystemMessagePromptTemplate.from_template(template)\n",
+        "human_template=\"{text}\"\n",
+        "human_message_prompt = HumanMessagePromptTemplate.from_template(human_template)"
+      ],
+      "metadata": {
+        "id": "XEsYhpKMFcud"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "fprompt=PromptTemplate(\n",
+        "    template=\"You are a helpful assistant that grades student responses ({answer}) to questions ({question}) about machine learning.\",\n",
+        "    input_variables=[\"question\", \"answer\"],\n",
+        ")\n",
+        "system_message_prompt = SystemMessagePromptTemplate(prompt=prompt)"
+      ],
+      "metadata": {
+        "id": "En5SM-ZHCT8l"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "chat_prompt = ChatPromptTemplate.from_messages([system_message_prompt, human_message_prompt])\n",
+        "\n",
+        "# get a chat completion from the formatted messages\n",
+        "chat_prompt.format_prompt(input = [q2, example1], text= \"The student should recieve an A-level grade.\" ).to_messages()"
+      ],
+      "metadata": {
+        "id": "W2zIxXxDFNIh"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "example_human = SystemMessagePromptTemplate.from_template(\n",
+        "    \"Machine learning is awesome\", additional_kwargs={\"name\": \"example_user\"}\n",
+        ")\n",
+        "example_ai = SystemMessagePromptTemplate.from_template(\n",
+        "    \"This student should reieve a C-level grade.\", additional_kwargs={\"name\": \"example_assistant\"}\n",
+        ")\n",
+        "human_template = \"{text}\"\n",
+        "human_message_prompt = HumanMessagePromptTemplate.from_template(human_template)"
+      ],
+      "metadata": {
+        "id": "b_r2Y7YXTa0R"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "chat = ChatOpenAI(temperature=0)"
+      ],
+      "metadata": {
+        "id": "R2XA1z27cl6V"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "chat_prompt = ChatPromptTemplate.from_messages(\n",
+        "    [system_message_prompt, example_human, example_ai, human_message_prompt]\n",
+        ")\n",
+        "chain = LLMChain(llm=chat, prompt=chat_prompt)\n",
+        "# get a chat completion from the formatted messages\n",
+        "text_dict = {'text': 'Please grade the following students response to question 2 based on the A-, B-, C-level grading scale'}\n",
+        "chain.run(text_dict)"
+      ],
+      "metadata": {
+        "id": "3WNMa70jVuf5"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "\n",
+        "query = f\"\"\" Please grade the following students answer: {example1} to the question ({q2}).\n",
+        "The uploaded json should serve as as examples of A, B, and C level answers. In the document, the\n",
+        "original question is printed, as well as examples of previous student answers that have recieved\n",
+        "A, B, and C grades (labeled accordingly)\"\"\"\n",
+        "grades = qa.run(query)\n",
+        "print(grades)"
+      ],
+      "metadata": {
+        "id": "naKuTxKa2-U8"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "## Conclusions based on Question 2\n",
+        "\n",
+        "Using different types of chains and example selectors made minor differences, but the results were still not entirely correct, and also inconsistent. Using a persona prompt as the system message was semi-useful, as was adding the grading criteria to the prompt. But it did not remedy our problem.\n",
+        "\n",
+        "Moving forward:\n",
+        "\n",
+        "1. Grading criteria helped, but for longer/more complex questions, more context may be needed (i.e., more specifics on what seperates an A from B, B from C, etc. The model did not perform as well in distinguishing between these levels/did not seem to understand the nuances of the grading criteria).\n",
+        "2. The model would likely perform better with a better example set/more context to be trained on.\n",
+        "\n"
+      ],
+      "metadata": {
+        "id": "C7kDC2pe7bNd"
+      }
+    }
+  ],
+  "metadata": {
+    "colab": {
+      "provenance": [],
+      "include_colab_link": true
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    },
+    "language_info": {
+      "name": "python",
+      "version": "3.10.6"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}

instructor_intr_notebook_grading_training.ipynb

@@ -0,0 +1,737 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "view-in-github",
+        "colab_type": "text"
+      },
+      "source": [
+        "<a href=\"https://colab.research.google.com/github/vanderbilt-data-science/lo-achievement/blob/adding_grading_levels_to_instructor_nb/instructor_intr_notebook_grading_training.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "collapsed": true,
+        "id": "brzvVeAsYiG2"
+      },
+      "source": [
+        "<a href=\"https://colab.research.google.com/github/vanderbilt-data-science/lo-achievement/blob/main/instructor_intr_notebook.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "WMKrKfx8_3fc"
+      },
+      "source": [
+        "# Instructor Grading and Assessment\n",
+        "This notebook executes grading of student submissions based on the examples provided in the [Wiki](https://github.com/vanderbilt-data-science/lo-achievement/wiki/Examples-of-great,-good,-and-poor-answers-to-questions) from Dr. Jesse Blocher. In this iteration, we use the Unstructured File Loader, which cannot proccess .json files (the preferred format). We are working on finding a file loader that allows .json. In this version of the notebook, the model has only been trained on Question 2 from the notebook.\n",
+        "\n",
+        "To train the model, we used 2 out of the three student example from each grade brack and inputted into a .pdf with clearly defined levels. Then, we used the excluded answers to test the accuracy of the model's grading."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "## Grading based on A, B, and C-level answers from previous students to Question 2 from the [Wiki](https://github.com/vanderbilt-data-science/lo-achievement/wiki/Examples-of-great,-good,-and-poor-answers-to-questions):\n",
+        "\n",
+        "**Question 2:** Why is machine learning so important for businesses? Answer this question generally (i.e. such that it applies to many or at least most businesses)."
+      ],
+      "metadata": {
+        "id": "ZTkNQ-dL5iO5"
+      }
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "### Creating .json file from case examples (Question 2)\n",
+        "The purpose of this cell is to create a json file based on the previously submitted, graded work of students based on the case file provided by Dr. Blocher in the Wiki"
+      ],
+      "metadata": {
+        "id": "TYlGEusr64kA"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Context: question\n",
+        "\n",
+        "q2 = 'Question 2: Why is machine learning so important for businesses? Answer this question generally (i.e. such that it applies to many or at least most businesses).'"
+      ],
+      "metadata": {
+        "id": "-kmMFUaLs_Q1"
+      },
+      "execution_count": 1,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# A-level answers\n",
+        "\n",
+        "A_answer_1 = 'Machine learning is extremely important tool for businesses. It can be used in a variety of ways, but most importantly, it can be used to identify patterns within their data that might not otherwise be identified by human beings. For example, it can understand customer behaviors, optimize logistics, and expand efficiencies throughout the business. Machine learning does not get tired, meaning it can work as long as you want it to. It can sift through massive amounts of data, that no human being can look through in an efficient manner. Machine learning can be used as a tool to identify anomalies when something needs to be checked to save or gain money. The predictions that companies gain from machine learning are cheap, accurate, and automate. These machine learning algorithms can be brought to larger scales to encompass the whole business and its operations. It is important to note, Machine learning is just predictions. Predictions to understand important patterns that could make or break a company since they understand the patterns of their business more. It is an amazing tool, but should be used wisely and carefully because if not, it can expensive, useless, and straight up wrong.'\n",
+        "A_answer_2 = 'Machine learning is important for most of the sectors in business. Overall, it gives the company of an overview about what would be the trend for their business industry, and analyze the customer behavior to help business segment their customers groups. Today, many companies have a vast amount of information generated by behavior, computer, events, people, and devices. This massive amount of data is difficult for human to handle, and even if human manages it, it is not profitable as human labor is expensive. Thanks to machine learning, companies can utilize their in-house or even third-party data to make something useful for their business. In medical analysis, for example, with human, it takes a very long time to find patterns in thousands of MRI scans. On the other hand, machines can detect patterns in seconds by entering data as long as the information is correctly labeled or trained properly. Another example would be segmenting customer group. In marketing department, the business could use unsupervised machine learning to cluster their customer segments to generate personalized contents that are relevant for each of individuals.'\n",
+        "\n",
+        "# List creation\n",
+        "\n",
+        "A_answers_list = [A_answer_1, A_answer_2]\n"
+      ],
+      "metadata": {
+        "id": "yQT6aExSr1dP"
+      },
+      "execution_count": 2,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# B-level answers\n",
+        "\n",
+        "B_answer_1 = 'Companies use ML models to improve different aspects of their business, like manufacturing, hiring, deployment, advertising, etc. The main goal is to improve productive and increase profitability of the company. The ML models are fed with company and externally available data to help the company optimize its departments and in turn become more financially successful/ productive. For example, using purchasing history, the company can predict who to advertise products to, to increase sales.'\n",
+        "B_answer_2 = 'Machine learning allows business to have automated decision, scale, predictive analysis and performance. Machine learning also helps a business have a data strategy. This is how a firm uses data, data infrastructure, governance, etc. to accomplish its strategic goals and maintain/grow their competitive advantage within their industry.'\n",
+        "B_answer_3 = 'The short answer is ML can help make decisions for businesses. To be clarified, ML does not make decisions for businesses. I mean it can, but people have not trusted ML enough yet and ML has not been that good to let it directly make business decisions. Business people only use ML to help themselves get intuitions of how decisions should be made and make predictions of results they might get based on their decisions. For example, if a business tries to launch a new product, it will use ML to test whether it will work or not on a small scale before it is introduced to a large scale. People called this step piloting. In this step, people collect data that is generated by using the pilot product and analyze their next move. They could tweak some features based on the feedback. If they think the data in their interest shows the product performs well, they might predict that this product will be successful when it is introduced on a large scale. Then, they will launch it.'\n",
+        "# List creation\n",
+        "\n",
+        "B_answers_list = [B_answer_1, B_answer_2, B_answer_3]"
+      ],
+      "metadata": {
+        "id": "KB1CmeRwtRvf"
+      },
+      "execution_count": 3,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# C-level answers\n",
+        "\n",
+        "C_answer_1 = 'Machine learning powers many of the services we use today, such as the recommendation systems of Spotify and Netflix; search engines such as Google and Bing; social media such as TikTok and Instagram; voices such as Siri and Alexa, the list can go on. All these examples show that machine learning is already starting to play a pivotal role in today\"s data-rich world. Machines can help us sift through useful information that can lead to big breakthroughs, and we have seen the widespread use of this technology in various industries such as finance, healthcare, insurance, manufacturing, transformational change, etc.'\n",
+        "C_answer_3 = 'As technology advanced, there are tons of new data generated and stored. All industries experienced this surge in data, including business. There is a huge amount of business data stored and waited in the database of each firm and they need solutions to utilize these data. Machine learning is a very promising approach for firms to puts these data in and output a meaning pattern or result that could help the firms with their existing work. This could turn into a working product or provide insights that could enhance the efficiency of the company’s workflow. With machine learning, a firm could either enter a new market with the new product or save time and effort with the meaningful insights. Achieving these possibilities with the data they already owned is a low effort but high reward action. This is the reason machine learning is valued by many businesses recently.'\n",
+        "\n",
+        "# List creation\n",
+        "\n",
+        "C_answers_list = [C_answer_1, C_answer_3]"
+      ],
+      "metadata": {
+        "id": "3diAz43othjc"
+      },
+      "execution_count": 4,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "Q_and_A = [\"Question:\", q2, \"A-level Answers\", A_answers_list, \"B-level Answers\", B_answers_list, \"C-level Answers\", C_answers_list]"
+      ],
+      "metadata": {
+        "id": "oAnrMSU6u9do"
+      },
+      "execution_count": 5,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "import json\n",
+        "from google.colab import files\n",
+        "\n",
+        "def save_example_answers(examples, filename='wiki_ABC_Q2examples.json'):\n",
+        "  with open(filename, 'w') as file:\n",
+        "    json.dump(examples, file)\n",
+        "  files.download(filename)\n",
+        "\n",
+        "save_example_answers(Q_and_A)"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/",
+          "height": 17
+        },
+        "id": "B16iYMEnri9s",
+        "outputId": "3e565b3d-804c-4b5e-acc8-efeb955c6c14"
+      },
+      "execution_count": 6,
+      "outputs": [
+        {
+          "output_type": "display_data",
+          "data": {
+            "text/plain": [
+              "<IPython.core.display.Javascript object>"
+            ],
+            "application/javascript": [
+              "\n",
+              "    async function download(id, filename, size) {\n",
+              "      if (!google.colab.kernel.accessAllowed) {\n",
+              "        return;\n",
+              "      }\n",
+              "      const div = document.createElement('div');\n",
+              "      const label = document.createElement('label');\n",
+              "      label.textContent = `Downloading \"${filename}\": `;\n",
+              "      div.appendChild(label);\n",
+              "      const progress = document.createElement('progress');\n",
+              "      progress.max = size;\n",
+              "      div.appendChild(progress);\n",
+              "      document.body.appendChild(div);\n",
+              "\n",
+              "      const buffers = [];\n",
+              "      let downloaded = 0;\n",
+              "\n",
+              "      const channel = await google.colab.kernel.comms.open(id);\n",
+              "      // Send a message to notify the kernel that we're ready.\n",
+              "      channel.send({})\n",
+              "\n",
+              "      for await (const message of channel.messages) {\n",
+              "        // Send a message to notify the kernel that we're ready.\n",
+              "        channel.send({})\n",
+              "        if (message.buffers) {\n",
+              "          for (const buffer of message.buffers) {\n",
+              "            buffers.push(buffer);\n",
+              "            downloaded += buffer.byteLength;\n",
+              "            progress.value = downloaded;\n",
+              "          }\n",
+              "        }\n",
+              "      }\n",
+              "      const blob = new Blob(buffers, {type: 'application/binary'});\n",
+              "      const a = document.createElement('a');\n",
+              "      a.href = window.URL.createObjectURL(blob);\n",
+              "      a.download = filename;\n",
+              "      div.appendChild(a);\n",
+              "      a.click();\n",
+              "      div.remove();\n",
+              "    }\n",
+              "  "
+            ]
+          },
+          "metadata": {}
+        },
+        {
+          "output_type": "display_data",
+          "data": {
+            "text/plain": [
+              "<IPython.core.display.Javascript object>"
+            ],
+            "application/javascript": [
+              "download(\"download_8ec6f24c-9653-4335-8c1b-766926434399\", \"wiki_ABC_Q2examples.json\", 5931)"
+            ]
+          },
+          "metadata": {}
+        }
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "## **Start here** to interact with model"
+      ],
+      "metadata": {
+        "id": "PpKRr5Vw4_0F"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "! pip install -q langchain=='0.0.229' openai gradio numpy chromadb tiktoken unstructured pdf2image pydantic==\"1.10.8\" jq"
+      ],
+      "metadata": {
+        "id": "UJi1Oy0CyPHD"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# import necessary libraries here\n",
+        "from getpass import getpass\n",
+        "from langchain.llms import OpenAI as openai\n",
+        "from langchain.chat_models import ChatOpenAI\n",
+        "from langchain.prompts import PromptTemplate\n",
+        "from langchain.document_loaders import TextLoader\n",
+        "from langchain.indexes import VectorstoreIndexCreator\n",
+        "from langchain.text_splitter import CharacterTextSplitter\n",
+        "from langchain.embeddings import OpenAIEmbeddings\n",
+        "from langchain.schema import SystemMessage, HumanMessage, AIMessage\n",
+        "import numpy as np\n",
+        "import os\n",
+        "from langchain.vectorstores import Chroma\n",
+        "from langchain.document_loaders.unstructured import UnstructuredFileLoader\n",
+        "from langchain.document_loaders import UnstructuredFileLoader\n",
+        "from langchain.chains import VectorDBQA\n",
+        "from langchain.document_loaders import JSONLoader\n",
+        "import json\n",
+        "from pathlib import Path\n",
+        "from pprint import pprint"
+      ],
+      "metadata": {
+        "id": "YHytCUoExrYe"
+      },
+      "execution_count": 18,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "from google.colab import files\n",
+        "uploaded = files.upload()"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/",
+          "height": 74
+        },
+        "id": "Wpt6qsmEw8WP",
+        "outputId": "4563fa62-5245-4115-ecb9-326353dba29c"
+      },
+      "execution_count": 7,
+      "outputs": [
+        {
+          "output_type": "display_data",
+          "data": {
+            "text/plain": [
+              "<IPython.core.display.HTML object>"
+            ],
+            "text/html": [
+              "\n",
+              "     <input type=\"file\" id=\"files-6f630631-16ee-49df-8de9-29bfa3ec5f7b\" name=\"files[]\" multiple disabled\n",
+              "        style=\"border:none\" />\n",
+              "     <output id=\"result-6f630631-16ee-49df-8de9-29bfa3ec5f7b\">\n",
+              "      Upload widget is only available when the cell has been executed in the\n",
+              "      current browser session. Please rerun this cell to enable.\n",
+              "      </output>\n",
+              "      <script>// Copyright 2017 Google LLC\n",
+              "//\n",
+              "// Licensed under the Apache License, Version 2.0 (the \"License\");\n",
+              "// you may not use this file except in compliance with the License.\n",
+              "// You may obtain a copy of the License at\n",
+              "//\n",
+              "//      http://www.apache.org/licenses/LICENSE-2.0\n",
+              "//\n",
+              "// Unless required by applicable law or agreed to in writing, software\n",
+              "// distributed under the License is distributed on an \"AS IS\" BASIS,\n",
+              "// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
+              "// See the License for the specific language governing permissions and\n",
+              "// limitations under the License.\n",
+              "\n",
+              "/**\n",
+              " * @fileoverview Helpers for google.colab Python module.\n",
+              " */\n",
+              "(function(scope) {\n",
+              "function span(text, styleAttributes = {}) {\n",
+              "  const element = document.createElement('span');\n",
+              "  element.textContent = text;\n",
+              "  for (const key of Object.keys(styleAttributes)) {\n",
+              "    element.style[key] = styleAttributes[key];\n",
+              "  }\n",
+              "  return element;\n",
+              "}\n",
+              "\n",
+              "// Max number of bytes which will be uploaded at a time.\n",
+              "const MAX_PAYLOAD_SIZE = 100 * 1024;\n",
+              "\n",
+              "function _uploadFiles(inputId, outputId) {\n",
+              "  const steps = uploadFilesStep(inputId, outputId);\n",
+              "  const outputElement = document.getElementById(outputId);\n",
+              "  // Cache steps on the outputElement to make it available for the next call\n",
+              "  // to uploadFilesContinue from Python.\n",
+              "  outputElement.steps = steps;\n",
+              "\n",
+              "  return _uploadFilesContinue(outputId);\n",
+              "}\n",
+              "\n",
+              "// This is roughly an async generator (not supported in the browser yet),\n",
+              "// where there are multiple asynchronous steps and the Python side is going\n",
+              "// to poll for completion of each step.\n",
+              "// This uses a Promise to block the python side on completion of each step,\n",
+              "// then passes the result of the previous step as the input to the next step.\n",
+              "function _uploadFilesContinue(outputId) {\n",
+              "  const outputElement = document.getElementById(outputId);\n",
+              "  const steps = outputElement.steps;\n",
+              "\n",
+              "  const next = steps.next(outputElement.lastPromiseValue);\n",
+              "  return Promise.resolve(next.value.promise).then((value) => {\n",
+              "    // Cache the last promise value to make it available to the next\n",
+              "    // step of the generator.\n",
+              "    outputElement.lastPromiseValue = value;\n",
+              "    return next.value.response;\n",
+              "  });\n",
+              "}\n",
+              "\n",
+              "/**\n",
+              " * Generator function which is called between each async step of the upload\n",
+              " * process.\n",
+              " * @param {string} inputId Element ID of the input file picker element.\n",
+              " * @param {string} outputId Element ID of the output display.\n",
+              " * @return {!Iterable<!Object>} Iterable of next steps.\n",
+              " */\n",
+              "function* uploadFilesStep(inputId, outputId) {\n",
+              "  const inputElement = document.getElementById(inputId);\n",
+              "  inputElement.disabled = false;\n",
+              "\n",
+              "  const outputElement = document.getElementById(outputId);\n",
+              "  outputElement.innerHTML = '';\n",
+              "\n",
+              "  const pickedPromise = new Promise((resolve) => {\n",
+              "    inputElement.addEventListener('change', (e) => {\n",
+              "      resolve(e.target.files);\n",
+              "    });\n",
+              "  });\n",
+              "\n",
+              "  const cancel = document.createElement('button');\n",
+              "  inputElement.parentElement.appendChild(cancel);\n",
+              "  cancel.textContent = 'Cancel upload';\n",
+              "  const cancelPromise = new Promise((resolve) => {\n",
+              "    cancel.onclick = () => {\n",
+              "      resolve(null);\n",
+              "    };\n",
+              "  });\n",
+              "\n",
+              "  // Wait for the user to pick the files.\n",
+              "  const files = yield {\n",
+              "    promise: Promise.race([pickedPromise, cancelPromise]),\n",
+              "    response: {\n",
+              "      action: 'starting',\n",
+              "    }\n",
+              "  };\n",
+              "\n",
+              "  cancel.remove();\n",
+              "\n",
+              "  // Disable the input element since further picks are not allowed.\n",
+              "  inputElement.disabled = true;\n",
+              "\n",
+              "  if (!files) {\n",
+              "    return {\n",
+              "      response: {\n",
+              "        action: 'complete',\n",
+              "      }\n",
+              "    };\n",
+              "  }\n",
+              "\n",
+              "  for (const file of files) {\n",
+              "    const li = document.createElement('li');\n",
+              "    li.append(span(file.name, {fontWeight: 'bold'}));\n",
+              "    li.append(span(\n",
+              "        `(${file.type || 'n/a'}) - ${file.size} bytes, ` +\n",
+              "        `last modified: ${\n",
+              "            file.lastModifiedDate ? file.lastModifiedDate.toLocaleDateString() :\n",
+              "                                    'n/a'} - `));\n",
+              "    const percent = span('0% done');\n",
+              "    li.appendChild(percent);\n",
+              "\n",
+              "    outputElement.appendChild(li);\n",
+              "\n",
+              "    const fileDataPromise = new Promise((resolve) => {\n",
+              "      const reader = new FileReader();\n",
+              "      reader.onload = (e) => {\n",
+              "        resolve(e.target.result);\n",
+              "      };\n",
+              "      reader.readAsArrayBuffer(file);\n",
+              "    });\n",
+              "    // Wait for the data to be ready.\n",
+              "    let fileData = yield {\n",
+              "      promise: fileDataPromise,\n",
+              "      response: {\n",
+              "        action: 'continue',\n",
+              "      }\n",
+              "    };\n",
+              "\n",
+              "    // Use a chunked sending to avoid message size limits. See b/62115660.\n",
+              "    let position = 0;\n",
+              "    do {\n",
+              "      const length = Math.min(fileData.byteLength - position, MAX_PAYLOAD_SIZE);\n",
+              "      const chunk = new Uint8Array(fileData, position, length);\n",
+              "      position += length;\n",
+              "\n",
+              "      const base64 = btoa(String.fromCharCode.apply(null, chunk));\n",
+              "      yield {\n",
+              "        response: {\n",
+              "          action: 'append',\n",
+              "          file: file.name,\n",
+              "          data: base64,\n",
+              "        },\n",
+              "      };\n",
+              "\n",
+              "      let percentDone = fileData.byteLength === 0 ?\n",
+              "          100 :\n",
+              "          Math.round((position / fileData.byteLength) * 100);\n",
+              "      percent.textContent = `${percentDone}% done`;\n",
+              "\n",
+              "    } while (position < fileData.byteLength);\n",
+              "  }\n",
+              "\n",
+              "  // All done.\n",
+              "  yield {\n",
+              "    response: {\n",
+              "      action: 'complete',\n",
+              "    }\n",
+              "  };\n",
+              "}\n",
+              "\n",
+              "scope.google = scope.google || {};\n",
+              "scope.google.colab = scope.google.colab || {};\n",
+              "scope.google.colab._files = {\n",
+              "  _uploadFiles,\n",
+              "  _uploadFilesContinue,\n",
+              "};\n",
+              "})(self);\n",
+              "</script> "
+            ]
+          },
+          "metadata": {}
+        },
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "Saving wiki_ABC_Q2examples (2).json to wiki_ABC_Q2examples (2).json\n"
+          ]
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# setup open AI api key\n",
+        "openai_api_key = getpass()"
+      ],
+      "metadata": {
+        "id": "jVPEFX3ixJnM"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "os.environ[\"OPENAI_API_KEY\"] = openai_api_key\n",
+        "openai.api_key = openai_api_key"
+      ],
+      "metadata": {
+        "id": "MO7VuGmrxVAr"
+      },
+      "execution_count": 19,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "mdl_name = 'gpt-3.5-turbo-0301'"
+      ],
+      "metadata": {
+        "id": "4Thxj6Gk1zVS"
+      },
+      "execution_count": 20,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "llm = ChatOpenAI(model='gpt-3.5-turbo-16k')\n",
+        "messages = [\n",
+        "    SystemMessage(content=\"You are a helpful assistant.\"),\n",
+        "    HumanMessage(content=\"\")\n",
+        "]"
+      ],
+      "metadata": {
+        "id": "Sgq9aVqpxZnK"
+      },
+      "execution_count": 36,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "file_path='/content/wiki_ABC_Q2examples (2).json'"
+      ],
+      "metadata": {
+        "id": "Ak7X_ZRba48F"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "data = json.loads(Path(file_path).read_text())\n",
+        "data = str(data)"
+      ],
+      "metadata": {
+        "id": "PKqQROVMc6BP"
+      },
+      "execution_count": 26,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Contruct Vector Store\n",
+        "text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)\n",
+        "texts = text_splitter.split_text(data)\n",
+        "\n",
+        "embeddings = OpenAIEmbeddings()\n",
+        "\n",
+        "db = Chroma.from_texts(texts, embeddings)\n",
+        "\n",
+        "qa = VectorDBQA.from_chain_type(llm=ChatOpenAI(model_name = mdl_name), chain_type=\"stuff\", vectorstore=db, k=1)"
+      ],
+      "metadata": {
+        "id": "i2NadtLlcxur"
+      },
+      "execution_count": 37,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# A level\n",
+        "example1 = 'The most powerful aspect of machine learning is its ability to automate processes. If the goal is well defined and repeatable, an algorithm can be trained to perform that task far faster than any human and often with more reliability. Because of this, businesses can implement machine learning algorithms to solve problems human labor was previously required for, whether that be mental or physical labor. Although implementing machine learning often has high initial costs, because computers do not require payment outside of maintaining their operation, in the long run companies can save money by either making their workers’ tasks more efficient or by entirely automating tasks. This increases the profit margins for those firms.'\n",
+        "# B level\n",
+        "example2 ='ML systems can help improve access to data while managing compliance. It helps businesspeople deal with data in a more efficient way, since there is numerous data generated per second in such industry. If they use some of the simple classification models with high accuracy, it will save them a lot of time on data cleaning and validation, which are kind of repetitive and time-consuming. For example, NLP provides an easy way for personnel to query business information, understand business processes, and discover new relationships between business data, ideas based on intuition and insight often emerge. Using models to do prediction helps people making wiser decision. Since models can handle way more data than human, newly collected data can feed in the model and get some predictive result as a reference to the decision makers. This is significant because in this industry, time is precious, and traders must decide quickly and precisely. A little negligence will lead to a big mistake, lose a lot of money, and even affect the company\"s reputation. Models can see patterns that are not easy for human to spot, which is also valuable for modify the way people doing analysis and interpret.'\n",
+        "# C level\n",
+        "example3 = 'The machine learning model (or one a broader view, artificial intelligence) is about prediction. According to the lecture, there are tree main innovations in it. Prediction is cheap, more accurate and automated. As a result, armed with machine learning, businesses could automatically gain much more accurate and powerful capabilities in forecasting, leading to a big savings both in time and money.'\n",
+        "\n",
+        "# Randomized list of answers\n",
+        "training_answers = [example2, example1, example3]"
+      ],
+      "metadata": {
+        "id": "CJxUs8lG12kd"
+      },
+      "execution_count": 38,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "query = f\"\"\" Please grade the following student answers: {training_answers} to the question ({q2}).\n",
+        "The uploaded pdf should serve as as examples of A, B, and C level answers. In the document, the\n",
+        "original question is printed, as well as examples of previous student answers that have recieved\n",
+        "A, B, and C grades (labeled accordingly)\"\"\"\n",
+        "\n",
+        "query_prefix = \"\"\" The uploaded pdf should serve as as examples of A, B, and C level answers.\n",
+        "In the document, the original question is printed, as well as examples of previous student answers that have recieved\n",
+        "A, B, and C grades (labeled accordingly).\"\"\"\n",
+        "answer = qa.run(query_prefix + query)\n",
+        "print(answer)"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "naKuTxKa2-U8",
+        "outputId": "fc10a157-071d-4a09-bf4d-315fec1d53e9"
+      },
+      "execution_count": 56,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "The first answer would be a B-level answer. It mentions the efficiency of using machine learning for data cleaning and validation, as well as the ability to handle large amounts of data and make predictions. However, it could be more specific in terms of how machine learning can benefit businesses in various industries.\n",
+            "\n",
+            "The second answer would be an A-level answer. It highlights the ability of machine learning to automate processes and save money in the long run, as well as mentioning the reliability and efficiency of algorithms compared to human labor.\n",
+            "\n",
+            "The third answer would also be an A-level answer. It accurately describes the main innovation of machine learning as being prediction, and how it can lead to cost savings in time and money for businesses. It also hints at the potential for machine learning to improve forecasting capabilities.\n"
+          ]
+        }
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "## Conclusions based on Question 2\n",
+        "\n",
+        "Moving forward:\n",
+        "\n",
+        "\n",
+        "1.   Train the model on the other questions provided in the case example\n",
+        "2.   Request more student answers from Dr. Blocher\n",
+        "3. Request a grading rubric from Dr. Blocher for the case examples, which may help the model gain more context\n",
+        "\n"
+      ],
+      "metadata": {
+        "id": "C7kDC2pe7bNd"
+      }
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "### Ouput for query with .pdf\n",
+        "\n",
+        "We also experimenting with using a .pdf file (before we figured out how to parse .json files). Here is one set of results.\n",
+        "\n",
+        "The first answer can be graded as a B. It touches on the benefits of machine learning such as improving access to data, efficiency in data management, and faster decision-making. However, the answer could be strengthened by providing more specific examples of how machine learning has benefited businesses.\n",
+        "\n",
+        "The second answer can be graded as an A. It provides a clear and concise explanation of how machine learning can automate processes and save companies money in the long run. The answer also acknowledges the initial costs of implementing machine learning but emphasizes the potential for increased profit margins.\n",
+        "\n",
+        "The third answer can also be graded as an A. It highlights the main innovation of machine learning which is prediction and how it can lead to cost and time savings for businesses. The answer is well-organized and provides a clear explanation of why machine learning is important for businesses in general.\n",
+        "\n",
+        "**Results:**\n",
+        "\n",
+        "The model did not perform as well as expected. It did not successfully grade any of the questions. The first qyestion should have been graded as an A, second as a B, and third as a C."
+      ],
+      "metadata": {
+        "id": "iJTiGX4aZ91_"
+      }
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "### Output for .json queries\n",
+        "\n",
+        "**First query:**\n",
+        "\n",
+        "The first answer would be a B-level answer. It touches on the efficiency and time-saving benefits of machine learning, but could benefit from more specific examples and a deeper explanation of how it can improve business outcomes.\n",
+        "\n",
+        "The second answer would be an A-level answer. It provides a clear explanation of how machine learning can automate processes and improve efficiency, leading to cost savings and increased profits for businesses.\n",
+        "\n",
+        "The third answer would also be an A-level answer. It focuses on the predictive power of machine learning and how it can save time and money for businesses by providing more accurate forecasts. It also mentions the automation benefits of machine learning.\n",
+        "\n",
+        "Notes: Same results as first query using .pdf.\n",
+        "\n",
+        "**Second query:**\n",
+        "\n",
+        "Answer 1: B-level answer. The answer talks about how machine learning can automate processes and solve problems that were previously done by human labor. It also mentions how implementing machine learning can save money in the long run.\n",
+        "\n",
+        "Answer 2: A-level answer. The answer discusses how machine learning can help with data cleaning and validation, which can save time and enhance efficiency. It also mentions how models can handle more data than humans and can provide predictive results for decision-makers. Additionally, it talks about how machines can see patterns that are difficult for humans to spot, which can help with analysis and interpretation.\n",
+        "\n",
+        "Answer 3: C-level answer. The answer discusses how machine learning can provide accurate and powerful capabilities in forecasting, resulting in savings in time and money. It also talks about the three main innovations in machine learning, which are prediction, accuracy, and automation. The answer provides examples of different industries that have implemented machine learning, showcasing its importance in today's data-rich world.\n",
+        "\n",
+        "Notes: Graded one answer correctky (answer C).\n",
+        "\n",
+        "**Third query**\n",
+        "\n",
+        "The first answer would be a B-level answer. While it touches on some important points such as saving time on data cleaning and validation and using models for prediction, it lacks depth and doesn't provide specific examples or applications for businesses.\n",
+        "\n",
+        "The second answer would be an A-level answer. It provides a clear and concise explanation of how machine learning can automate processes and save companies money in the long run. It also acknowledges the initial costs of implementing machine learning and how it can increase profit margins.\n",
+        "\n",
+        "The third answer would also be a B-level answer. While it mentions the three main innovations of machine learning (cheap, more accurate, and automated predictions), it doesn't provide specific examples or applications for businesses and lacks depth in its explanation.\n",
+        "\n",
+        "Notes: None of the answers correct\n",
+        "\n",
+        "### General comments:\n",
+        "\n",
+        "None of the query results were consistent. There were also intsances where the output would be something like \"As an AI language model, I do not have the capability to...\". So more training and prompt engineering is certainly needed."
+      ],
+      "metadata": {
+        "id": "FR4YMpPseoTE"
+      }
+    }
+  ],
+  "metadata": {
+    "colab": {
+      "provenance": [],
+      "include_colab_link": true
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    },
+    "language_info": {
+      "name": "python",
+      "version": "3.10.6"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}

instructor_vector_store_creator.ipynb

@@ -0,0 +1,489 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "HpzC9I3uDBud"
+   },
+   "source": [
+    "<a href=\"https://colab.research.google.com/github/vanderbilt-data-science/lo-achievement/blob/main/instructor_vector_store_creator.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "hbthVGIgDBug"
+   },
+   "source": [
+    "# Creating a Shared Vector Store (for Instructors)\n",
+    "\n",
+    "This notebook is for instructors to create a *vector store* which contains all of the information necessary for students to generate their own self-study materials using large language models."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "NpYSvDgzDBuh"
+   },
+   "source": [
+    "## Using This Notebook\n",
+    "This notebook is split up into a bunch of *cells*, which contain explanations / instructions or code. You should go through the cells in order from top to bottom.\n",
+    "\n",
+    "When you encounter a code cell, hover the mouse over it and press the 'run' button (the circle with a triangle inside of it) to execute the code in the cell. Wait until the green checkmark appears to the right of the run button before proceeding to the next cell.\n",
+    "\n",
+    "Run the following code block to test it out. It should display `4` below the code.\n",
+    "\n",
+    "*Note: Before running, there may be a pop-up saying 'Warning: This notebook was not authored by Google'. In that case, click 'Run anyways'.*"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "ACE73ZmZDBuh"
+   },
+   "outputs": [],
+   "source": [
+    "print(2+2)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "dTrRB_2rDBui"
+   },
+   "source": [
+    "## Importing Libraries\n",
+    "Continue running the code below. Ignore any warnings that pop up."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "0MHRWInBDBuj"
+   },
+   "outputs": [],
+   "source": [
+    "# install libraries here\n",
+    "# -q flag for \"quiet\" install\n",
+    "!pip install -q langchain openai unstructured tiktoken deeplake trafilatura justext yt_dlp pydub"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "2FpSOOlEDBuk"
+   },
+   "outputs": [],
+   "source": [
+    "# import libraries here\n",
+    "import os\n",
+    "\n",
+    "from google.colab import files\n",
+    "from getpass import getpass\n",
+    "from IPython.display import display, Markdown\n",
+    "\n",
+    "import openai\n",
+    "from langchain.docstore.document import Document\n",
+    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
+    "from langchain.embeddings import OpenAIEmbeddings\n",
+    "from langchain.document_loaders.unstructured import UnstructuredFileLoader\n",
+    "\n",
+    "from langchain.document_loaders.generic import GenericLoader\n",
+    "from langchain.document_loaders.parsers import OpenAIWhisperParser\n",
+    "from langchain.document_loaders.blob_loaders.youtube_audio import YoutubeAudioLoader\n",
+    "\n",
+    "from langchain.document_loaders import WebBaseLoader\n",
+    "import trafilatura\n",
+    "import requests\n",
+    "import justext\n",
+    "\n",
+    "import deeplake\n",
+    "from langchain.vectorstores import DeepLake"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "TIFQUXspDBul"
+   },
+   "source": [
+    "## Setting Up API Access\n",
+    "Much of the following code rely on certain *APIs* (application programming interfaces) which have limited access. You will need to get an *API key* for each of those services which will be inserted into the code to let the service know you are an authorized user."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "vKXV9FubDBum"
+   },
+   "source": [
+    "#### OpenAI"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "GG9GwadDDBuo"
+   },
+   "source": [
+    "First, you will need an **OpenAI API key**. To do this:\n",
+    "1. Visit [platform.openai.com/account/api-keys](https://platform.openai.com/account/api-keys) and sign up for an account.\n",
+    "2. Click 'Create a secret API key', and give it any name you want.\n",
+    "3. Copy the newly created key, either by right-clicking and pressing 'Copy' or using the keyboard shortcut -- Ctrl+C on Windows, Cmd+C on a Mac.\n",
+    "\n",
+    "Run the following code cell. You'll see a blank text box pop up -- paste your API key there (using the shortcut Ctrl+V on Windows, or Cmd+V if you are using a Mac) and press Enter."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "Dv3l_ZKiDBup"
+   },
+   "outputs": [],
+   "source": [
+    "OPENAI_API_KEY = getpass(\"OpenAI API key: \")\n",
+    "os.environ[\"OPENAI_API_KEY\"] = OPENAI_API_KEY\n",
+    "openai.api_key = OPENAI_API_KEY"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "1Xbl8szQDBup"
+   },
+   "source": [
+    "#### DeepLake\n",
+    "\n",
+    "Next, you will need to input a **DeepLake API key**, found in the DeepLake dashboard at [app.activeloop.ai](https://app.activeloop.ai).\n",
+    "\n",
+    "1. Click the link above and create an account.\n",
+    "2. After making an account, you will be prompted to set a username. Once you have set your username, copy it, run the code below, paste the username into the text box, and press Enter. (This username will be shared with students.)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "rfzGVvm-DBup"
+   },
+   "outputs": [],
+   "source": [
+    "DEEPLAKE_USERNAME = input(\"DeepLake username: \")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "o9_N6ychDBur"
+   },
+   "source": [
+    "3. You should then be on the DeepLake dashboard. At the top, click 'Create API token'. You should see an empty table with the columns 'Name', 'Expiration date', and 'Token'.\n",
+    "4. Click the 'Create API token' button ![image.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAJYAAAAwCAYAAADzRIMRAAALoElEQVR4Xu1caXBT1xX+nvZdssFm8+DYgGPWAVMggFkyMWuJyYBDgRICDZ1JO6HJkNKkAYakEHcgSxsyQybDUkqLW9bWCVsh7GFJ2NywhuASJyQ4GCxZaF973nuWbFmSbWHJjjXv/tLy3r3nnvPdc7577nmP8VOD0AQNxFkDjACsOGtU6I7TgAAsAQgJ0YAArISoVehUAJaAgYRoQABWQtQqdBozsLhNpM8J193P4a78FO6q8/CZr8NruQp4BIW2Ww1IALGmD0S6XEjTBkPaOR+y9KGASA6GYWKeVvOB5ffCY70D+5cb4aooBSQqyDuPhTj9J5AaciFSZ0AkVccsgHDDj0MDPrcVPuttuE3kJO6eg7PyKDkKG2SZU6F89BeQqLvQVk/cbGGbBpbfB5/bDNul92AjUCmynoay12xIO+Y1exDhwvapAfe9C7B/VQLHre1QEbhU/V8k56EjgImanFCjwPJ7HXDeOQZr2VuQ6nO4jiWGR5vsVLgguTTgMX3JORZ3zQ2oBy6GvMsYcl6KRicZFVh+coO2LzfBdnE5NMPfh7LHzOTSljCbmDVgL/8nLKcXQjXoDfJg88AQHYrWIgKLA9XVD+Ao3wbdqLUU9gbHLIRwQ3JqwH3vPMwnfg1FjxlQ9flVVHCFAcvvdcJ2fQMcN/4K/eObhdCXnPho0azY0FhzZC4UOc9ClfschUV5WH+hwCKi7vjuEB4cnQ3DxP2Cp2qR+pP7ZtZzmfZPhHZsCRTdnggj9CHA8rlMMB4ogqr3AoFTJTcu4jI7lnPZrq1HyvgdEMkMIX3WAYvyVJYLK+Gz/cDxKqEJGmiOBli+JVJ1giZvaUieKwgsj+U2qj8aidTJBwRe1RyNCtdwGmD5VvXe8UgtPAmJJiOoFQ5Y7DGN5cIK+F1m6Ia/LahM0EBMGjCf/i0YmY681rLg8Q8PLEqEVpeOhG70OiGjHpNKhYtZDbAZevPxXyJ16slg4pQDlvPOcVjOLkGHwhNx1ZTH/hmc5VfA1j4zkmzKfdDZojSuQySoMyNsN3bB59JCnjUDkY5AHd9sgMdc/3A2HfLMUZBo9WB/9aMctsvHwGjHQpWZHRc52TG9jkFQ5uSh6UOVuAzZ7E7ufzQKmiFvUlZ+NHcPByzLxWKCnR2aoSua3VFjF/rxA8yHhsF5qyL0MlEuVPnHoclJi8s4DTuxX3+JzrW6QzNpEVqCX4/pDzDuWM4tCGnPM0gZOyxMXtNeBq7vw6ch6VYC/aRZBK7NqF7/LJguJejw01lhFz6MrOyYbvNSpMxcASpG+FE1y+fLSFlKaAa9Vgcs44HplOyaC+UjU+MgrAvmoyo4bnoh6bYe2sfnQqqQ0jnTFpj3zIHXVQTDrO2QhefUWjy25VQmnRgMgWHBDsha0Jv5mJhOHZ6AuNNBeKuegeGZzZA1ONjngTUZ2qfXckb2oxLWY4/BfRdQj/VD2bNxYD2MrD9mYNm/LqWk+mZKPeysA9b9HX2hL9gVl92gx1wM47YlYDp+gJSnnkd9e/hst+CXZnHh0HO/GKZ9f4as71p4btGpeef9MIzoD49lL8yHX4Dn3i0qEMqBrMdaaPPJyJy4NlgvvQD75b9QiQd9lQ2CIm8rtP16wXpuJP1+Cn62JkxBYSnnGPRDc6ky4xIeHJ4N153LZHw1JJ3XQjduLiRRKkB8vl0wbp4OpJVA03sPag5vgXKEG9o+oT6CB9b0EBC7KxfBtPtPkOZ+AV3+xageyxJFVq/tIMxHXoan6hIoV00lScW0MH8PWW01UiiwXLCcyYf9phOqEaegzmZIN/PpbHcbpwORZjY0BZugSOV9d80hhkL3ciqDOQ/Hpd2g7BKk3UuhKygMsdHDrkcuG//JNHQoulIHrLubGHT8uSUu9VTO60+i5tPdUD5GxugX3WG7q16BqXQ1/KJU8gzjIc9eDWXWSRi3zoJf+xLUg2dRAeGHsJ/dyBmKA92Dd2DauQ6S3r+j+iALXF+9SADNgGbit2AsL8H6xXt0TwYkGVSklrka6t73CeSD4PX8DKqhL9OO5d+wnSom0G9HyqSiiDzFeXMmao5uhXq0G6qcz1C9JR9+zToipgs4hQW4TTiwbERg+9CqreCAqO5TEhVYbBhsKKsqpxLGnUOpYDKXFttiiJiTcF7bSIuEQl8RhT5aCHXAWgbHuVGwlZ0lPd+Gpl9XWM/0Im/thbzvGsi7+2E/Xwj3/ZnQz/gH5Er+XjZ0izstgSwrHW5Od52gHlcJdebDwqnuPrae694WDdLn8U8Tchzr7noG6Qvi83ih9VweldlcJOX6aZVHFzgALEU9AHL3ftEJ+ln7OGWwzXpxOKz/7Y2UeQQw+u5zV8Frs5GFDfA7ilFD4FTUjtUwvLj+Nwemw8c44KlqUyzOiudgPniNQtgpKPQN5TPC9FEqXNXzg+NxYfGrPOimn+UuVqTw90TjWIz+daRMWw6ROLZQ6LwxDTXH/8UBWp3DL8gAyAMeMwAs5cAy2GjxynLPQ5+fR574Yxg3FUL0SB0f9PkOwFgyAZKcb8lzZ9QDJc/PAt41oLuWQwuoj6O4AyugIPkAfkLRWhBY9QAYzVhAb2ievAqp6A2YPn6dvJyadplu6lpK4LIGQdwQWPz3byKKEAn4AdIu7r4N2mFUlkvNW/MuHhxYAwmReLYFiDwvK+8duRUqzSYvSUWQWQMJVLQAmiDvkWUdCMP80iCfC/QhYT12fn8ezMThqHMw8oUwzFwDKY0V9P4RZiqu3Tw05GeR9N9ScIUDK46hMMBRvMw06It2Ql6vWtlZ8S48XgpL2d2IR/ChsP6KefBpV9ivD4B2+oeQhOynJVSP3Q2WwwwdkteR6YarrqGxHFcmwnz6GlRjTkCeHuqRRarMsNRHY0BkFAs5vRvmkDGDHiuUY9U3TKzA4mUtCwlNAePL80zQ5elrvSQDaa9n4CnfDFGnEhhox8n4NqB64wKIs3dDM7hfCD4YaRpVkavCPVYE/bcEWBFDYTzJOyucjXiWhVw15I9D0XcRZF38xD3egruc8mRqnjP4q8OB5a5aSh7pTTAd1kA/4XniFSbKBa0kYv82lF2ktNskYH1dCP3TpRxg+bBG5dK1Xs9eNgYPzlVBNW475Kk5xBl38pyNQKGbsAoyrRP2r1fRsyCLoOkTmvLw4yTHp3x4Hropr4bwL+fVKbBduczpPRCqIpH3WIDVmKzaJ4gPoowW0lPw2kN5UiDd4KrVMR8Os1Czn0L49wOgGl0KVc+u8FbvowNiKzQjZ3PkPNEeKyJ5j2+6gVevo+JVWE6QER116hanLYG2YCW3y4nmiu035xMoN/G7O7ZJ+tKu5wiX+/La/kYEdy7xLDbhqgSjzobfeCUIrAAw2R2V5JGDSC0ogKvyjzB/8lqdHKIMyPvTbmhIHpfIDLQAn5HVhp36IPE63ofx77/hf0rjibw5wq4wFmBFlfUgyeqs7YkWprbgP9yiYlsoOIgP1oJJM+EMFJ0vo2bPZEp33AuKIU5bDf2UxQ2Ify3HirPHiphuiHeCNETBzu8oW+yhE/Dw0FP/utDPLtoB3iFSaghmsuv+5/+DtCsklB9r2PxE7qkAFmJ9WojX8dgr6CxUFfZ7dBkS/09kWRube9My8X3aYtR30/02dUXEBGmijnSaEkb4P3k0EPFIRziETh4Dt8VMoh5CC2UzbWGO5BkzatkMO0Wh0C95DN2aM2m00I8TRChNbk17JM1YTZYmszMVHqZIGnu3ykSa9zAF57WEx79axSJJMEhMj3/xEVF4YDUJ7J7QKcT+wGqtOMIj9gm1S7vu/KEfsQ/MWngpSLu2f0KEb/FLQYLgorDIZuWtZauE1xglxFTto9PQ1xi9wj0wEel9DfVnI7x4rX3Ytk2kTNiL10JmI7wqsk2M21qDtv6rIhvMTHi5bWuZupXHabOX27byPIXh2rcGmuZY7Xt+gvRtpAEBWG2k+GQfVgBWslu4jeb3f6EJmcthjHP3AAAAAElFTkSuQmCC) at the right of the page, choose a name for the token, then click 'Create API token'. (You do not need to change the expiration date.)\n",
+    "5. Afterwards, you should see the table look something like this:"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "ivyqa05xDBur"
+   },
+   "source": [
+    "![image.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAsMAAABrCAYAAACSXI/oAAAgAElEQVR4Xu3dCVhN6QMG8DctSiZC9p0xzFgG2ZUlFMLIkkGWwYx9+BvZMxhjzSTTDJM9JoPIGMvIln2isa9DlsgMkZSi7f7POffe3K5KqZN7u+99nnlmps7yfb/vfPe85zvfORkphA/4oQAFKEABClCAAhSggAEKGDEMG2Crs8oUoAAFKEABClCAApIAwzAPBApQgAIUoAAFKEABgxVgGDbYpmfFKUABClCAAhSgAAUYhnkMUIACFKAABShAAQoYrADDsME2PStOAQpQgAIUoAAFKMAwzGOAAhSgAAUoQAEKUMBgBRiGDbbpWXEKUIACFKAABShAAYZhHgMUoAAFKEABClCAAgYrwDBssE3PilOAAhSgAAUoQAEKMAzzGKAABShAAQpQgAIUMFgBhmGDbXpWnAIUoAAFKEABClCAYZjHAAUoQAEKUIACFKCAwQowDBts07PiFKAABShAAQpQgAIMwzwGKEABClCAAhSgAAUMVoBh2GCbnhWnAAUoQAEKUIACFGAY5jFAAQpQgAIUoAAFKGCwAgzDBtv0rDgFKEABClCAAhSgAMMwjwEKUIACFKAABShAAYMVYBg22KZnxSlAAQpQgAIUoAAFGIZ5DFCAAhSgAAUoQAEKGKyALGH41vbJWHKxFeZ4dEQxDdpXIV6Y+FdLeI+xNVhwVlz3BcTjd96uyDQFNUIZuHw7Bx3Lv3v5Q9eNwuHiczDBWbNXZH97ydG7MX/GQ3RfPAQfm2V//ayuocAJLB16Dm1WjkS9rK7E5ShgAAJJD7Zgxsw/8TidujYfthJfNEkfgX3KAA4OVlEvBWQJw6fnNsbna56h7oAA/OZRB6Yqmritrmi9wxUhfi56icVCG4aAePwOP/8lAj0dNSpsjCKly8JKfTC/A8W7hmEFLsDDfhRK+hzFmDpAXoXhFPwKt6r7MCBsLTQl0qu6aDY3eT0CPWq+gwxXoYB+CSgSn+HhvzFIFop9wccFPimLsGJMNakSlsUroFih9OuTnT6lXyIsLQX0W0C2MPxFSA18fCcCLdYGYWx9Y0lJOwyLXyhnD+zElUgjWFWyRwe7ijBXeb54cg+vLMog9uLvOPJPAdRq2wUNbGKl5a9GlkQTF0dU1/jCibkbjH1Hw5FYvB46tauTo9Ci303K0udUQAx2o65NTveiTREXiQfPTFGqbBHVRV4CoiL+g6JoBVibCifIx0Yobn0f+wPOIrpQZdg7tUR51XH6KjoCUbBB6SKmUP+35bMT+DvqU7T6tAjS7w/C9h8exuyes1B81mYMbVRSWD8RkeHxKFyhuKq/JODJhf3Yc+EpTLSOf+mkLZSphHUUTuw9igcv3uw7ml6vos9h/85LQtlroE3Xm/imhmYYfr2fAhp1E+uyf143KRD8Mq5+qk1K/B0cUe2zQZe2qCXUmx8K5DeB9C4E1cf+/ReFUb2lE5pWtpCq/WYYjsPj8CdItiotfS8Ar/tYRudExc0gqa8Xqd4OTk1Lpw425TdX1ocCeSkgWxgWw0RAt83oNssG3qHL0FK4nasZhhUIwbTWo3D5077oZZuCgytW4b8WqxA4v5nUuf0GlsWSqw3QsEtXfHjXF2tDyqLex0CFxp2Bw944bDYJB7a6wUr48ji5oD0mBDWE66AGMLmyGb+GNIXv7qmy3kLOy0bivvJWILMwnJIYipmOQ/F8+EEs7W2N8K190OvHBlh/2B2VLnvAwTUIZpVqoH3X5nh5fB0CL9bFokPL0aEooHnSlPZxoCgKm5ZCuy8XYkqP8Az6QzkETJ+AnwOvwqx2M7TpNRuTexzSGLEVjv+5bTBsZyX0G9wZBf72EY7/TzBvny862QCvtMt0eBX+iBqIHXtHQXvGR+z52ejWdwcKt+8Pl4q3seXkFUSHVsD0sDXSyPCx2c3gfqwZXAc0wMujP2HThbbwDf4O1rsmY7L3TvyjqIvmLfrAY243FBa29dmwo6jvOgh1Cl9EwKq/4bBiT+qFcd62KPdGAfkEtMNwUqQ/BjjMQ0ybwehV6z9s/WUnzHtswsapdWCc5m6Lsu9OvjYWm1e7oqTpI2wc6og1sb3g6lwRUUIf2xv3DQKEO6nixCrxnOgT1gBlGzqgm9A/160LRvOFZ/GdYwH5KsctU8BABGQNwyF+ztgxug4WR8/DDqFDm2tNk0hMTISpqXK0SDxpO31ljhXHpqKGquOHdLiPZf0KQIFgjG/4AxrvDURf1QlevWyVu/Pg3DMac4/Ph61q/uSVJa0wO3kDNk2sYCDNyGrmpoByms/TNJs0UpTD2MBgaZpCUuQ69O8UhJ7r7bGx7wkM+3NtavB0HGaMn0/MRC1p7QRsHfExNpT5Q5o+oB2GpzzyxJ6ldqkjOxn1hw+1pkloji61FY7/Tp3CMfXsT2gjHf8J2D+pEebGe+OQdyupX2mWKePbtFFY7lofh22Ppvab5PiVcP3kOIapwrBC6K9JQn8Ve6z21A3NuikQhnmOrlBMP4Vpdsq7QolCObsONYdv0Pg3Qnhuth23RYG8FkgbhpV9fmvVQ6n9KOnJz+jZfC96HNiBfuWVU4/cwnxRZG5ruJ8bBr+Ng1BZ6LvPd38Bp5X2+GPbINWzNuK2miC0UyjmdSkghWH1OVGsI6cm5XVLc3/5WUDmMOwCcSRtqoMbYr86joUFh6fOGVbgEbaN7465wa9gYlEcH9e2xLXLtlivEYb/7hyBH3orT7wz7ZegxRHl3EXN4FxeCNgNFsahZd1Sqe2U8vwqrhacgJOcm5yfj13Z6pbZyLB6p+H+PeE04zrazDqCH/tZp3tBJ/5Q826IdhjWnGObWX/ILAzbpTMP/9X5yegwygq+Ql8SR6s1LzIzCsPa4VYsu/ay93ePQt/pJxFXoBBK1v4EZhevwmHtEekCQbNu4nr9qy3FizZ1hUkhCiWZ0TNcP/oBpl9TjjLzQ4H8IpD2QjDt/H5lHROwun8VXOz6AJ69hb5RdQ+qD7mNgM22+OW0F1qoBnHE7QwLqgbb6papNAkPQvG8mb90MS2GYfU5kWE4vxw9rIeuCMgehsWKxp73gLPrcTj2L4Pt112kuZjxQUNhv7A2tgeNk0aKYs9MROdx1tIJXD0ynNUw3NK/FQK8usBEQ9XIogTKl1DO0+KHAtkReFsYFoPrhoEO+CnMEmaV3bFddRtT++6GuM+I1c7ocewrnFzd5Y2RYc0wnFl/eFsYbhXgghP+rqkjzOK2WnnWx+/CVAjrbIXhEbBeehz/q6/U0gzD7bELw2t7o/HOPzG0ivC7xFOY1NodFZdnFIYD0WG/Jxw0OyUKwSZ1nnN2WoTLUkB3Bd4Mw2n7kQL3sLizI6LHXsVsRyEMV5+BO81Ho1WkD6433Z76kLm4nelP5mLl+LQPoZqq5hMzDOvuMcCS6b9AnoRhkenqMgd0XxYOqybzpTAcF+gG+/UOOCjcErISfq85mpWdMFwt2h9u9r5oqvGgXvzNq/ivYi3p1hM/FMiuwNvCsDgqPGh3X2zxs0Ngv044322fNH9YDMPtet/GxMN+6CpM51HOL+6Dx25/Y/ngDzINw5n1h8zCsEP0L+jd5De09t8nzccVg/qa/q1x0PZPbBhX4Y3pR5k9zb5rfA3Mj5wvTWkS5yg+Ey5Qu/SOlOYMt8dmDP5wC5xDtqCXMBCugHDHp8V4lE0nDEN4TNBvYHNsr7Y59USvSLyBG3eq4qMP06Tj7DYNl6eAzgloT1c4Mq0ept2bmdqPYoU7NZ36v8D3wrMzzc1ev6HFQTW32GZacOr3R/vP72HyAeW0K/Hz39WrMK1VK3XOMEeGda75WaB8IpBnYVg5mtYKy1LmSmFYDAqePYZgy6s6qFuhIBJe3hFOlm2zPU1CDM4Pgr/B4DH7kFyjCUrFX8DN2EaY/euPcCqXT1qJ1chTgczmDA9O8UCXwfcwUTVPWHxYZmAHXzRZFYSvzGahw5dhqFUuDGfCEpAcH4siTb7HttXKcJnZNInM+0OCNPd+8hErVO24GDsXPEjzyjPx+B8wcg+iLCxglJCAUm0WYsNSJ2mf2qPVmYXh5MgDmNR3OHb9WxSWZgqUammLAr+/wmhpznACTi3rglFrjPBxg/IomPgSdy7dQzfVNInnwmh0yzEhKFTcGV7Hv0cjYVuT3cbiQPzHqF/+BS5fjkNrj02Y3710nrYld0YBuQW0w7B44ffzoJ746WJBmJslIcmoKsb8shlDhItV7f6nvmvqIl3MJuPS6gEY7BWGMnXrweT+aTwtMQBL1/0P9YSZExwZlrsluX1DFpAlDGcHVHyFWoxJGdVrZbKzpvayyldcvbJUv6ImJ9viuhTIvoBm8Cz3jsd1xv0hTut1atrlU76iyTiTd5xmtUaZ9ck3Xy33eqvK1yFWTPOOVfG1a49fWGq8ii6rpeByFNBvgXc/9nOvL+u3IEtPgbwTeO9hOO+qyj1RQF6B9OYMy7tHbp0CFKAABShAgZwKMAznVJDrU0AlkPzsLHYfMUarrnWlefD8UIACFKAABSig+wIMw7rfRiwhBShAAQpQgAIUoIBMAgzDMsFysxSgAAUoQAEKUIACui/AMKz7bcQSUoACFKAABShAAQrIJMAwLBMsN0sBClCAAhSgAAUooPsCDMO630YsIQUoQAEKUIACFKCATAIMwzLBcrMUoAAFKEABClCAArovwDCs+23EElKAAhSgAAUoQAEKyCTAMCwTLDdLAQpQgAIUoAAFKKD7AgzDut9GLCEFKEABClCAAhSggEwCDMMywXKzFKAABShAAQpQgAK6L5BhGH769Knul54lpAAFKEABClCAAhSgQA4EODKcAzyuSgEKUIACFKAABSig3wIMw/rdfiw9BShAAQpQgAIUoEAOBBiGc4DHVSlAAQpQgAIUoAAF9FuAYVi/24+lpwAFKEABClCAAhTIgQDDcA7wuCoFKEABClCAAhSggH4LMAzrd/ux9BSgAAUoQAEKUIACORBgGM4BHlelAAUoQAEKUIACFNBvAYZh/W4/lp4CFKAABShAAQpQIAcCDMM5wOOqFKAABShAAQpQgAL6LcAwrN/tx9JTgAIUoAAFKEABCuRAgGE4B3hclQIUoAAFKEABClBAvwUYhvW7/Vh6ClCAAhSgAAUoQIEcCDAM5wCPq1KAAhSgAAUoQAEK6LcAw7B+tx9LTwEKUIACFKAABSiQAwGG4RzgcVUKUIACFKAABShAAf0WYBjW7/Zj6SlAAQpQgAIUoAAFciDAMJwDPK5KAQpQgAIUoAAFKKDfAgzD+t1+LD0FKEABClCAAhSgQA4EGIZzgMdVKUABClCAAhSgAAX0W4BhWL/bj6WnAAUoQAEKUIACsgnEx8fj5cuX2dq+ubk5LCwssrXO+1yYYfh96nPfFKAABShAAQpQQIcFbt26BfGf7HyqVasG8R99+TAM60tLsZwUoAAFKEABClAgjwXUYbhRo0ZZ2vPp06elIMwwnCWurC4Uh8fhz1CwdFlYmWZ1HS5HAQpQgAIUyH8CN4+ux6m7BdKtWLUW/dGsivJXisRn+OevAwi5Ey/9f+k6XdGqnhV4Gs1/x4QcNRKnRkRERKBs2bLSv8VA3KFDhyztat++fe8UhqOiomBtbZ3hPsQyyTX1Is9Ghl9d9kC7LuGYHrYGjlniBJIi/THE8VuEKixRu+dGbJpaK4trpr+YAhfg0aorYkfdwQ+9lV8WD/9NRrEKxWGeoy1z5fwqcGv7ZCy93xPeY2zfWkVx2d9SvsbUHmXeWDY58gA85xxCnYnfoWN55a8jghfAc19NTJzbDaUz2boi8QGOrvKGV8B+hEcVw0ftBuJbj/6oXuj1SvEP/4TXpAUIuPIcJWv1w5QFY2BX1kS1QALu/eWLZYu34dDtaJSo7Ighc2agVy3lUZ8SfxlrZk/Hr0HhiDEvjSaf/Q/Tv26LUjxrvrXNuQAF8lpg67g6mH+soLDbZMQ/f44U82KwNFNIxejkcQazuwKx5z3R94vluGX8ERrULY2CRlG4cfIykquPhu/m0fjYLK9Lzf3pm4AYTMURXnE0+OnTp7KHYfX+xPBdu3btN7hiYmJw5swZVKxYUZYRZ50Ow2cXNMeXp0fgwFY3WOXCkaQdht8loOdCMbgJPRI4PbcxRl2bjBA/l7eWWlx2bvJ6BHrUTLNsSvwpTOvshm33y2J0QDDG1AGenZmCbv0D8KzEIGw5NhU1Mtx6DDYOboTVRv/DHPdOKG/5EH/OGoHl9wdjx95REHN1cvQWfGE/B6bD12GmswVC14/H7E3V4R26DC2Fk97jbX3RYaElxi6aiHaVLXD/8DSM/+4Jhu/dhYFVzmKa3QDcajkHU0Y2hHXcBayaOAlHyi9F0E8OHEV6a6tzAQq8HwHt85m6FOL3wWD7qYjp+hv85zRIHehJSQzFTMfeOFxuEXYI32fF3k+xuVc9EcjrMCyyXLp0KXU0WjMQq4Ow+FCeGM5NTNQDPbmHKWMYjsOtY3twUrhFI96eaWayGE7pjAzH3A3GvqPheFWoMuydWqK8NNqVgKiI/3DC2wU+KYuwQkgPNqrRW3E09+yBnbgSaQSrSvboYFcxtbMr4iLx4JkpSpUtknoSfxUdgceJ1ihfwgKaXx5LnCMRdup7uA19iFHBC+FgVRqli3AoLPcOrfyxpbRhOA6R4fEonHonIQHPIiJRwEY5hSe9MKzAv1jeuw3215+JGnt8UNbnKEaU+gW9221FsylNsNvHAisyDcPCHYy4OKF/FHp9Uktai88/2oKOwbswSEjDV5a0wvDrU3BghZPquI/Cctf6ONbsFDaME8ec4xAfVwgWqSPJCVjdvwoO2oZKv09MTISp6etj/9X5yegwygq+bylX/mhh1oIC8gqId4VmjZyCgHPPYFLsU7jN74jbEy6gZ+gPaCPtOgGX/L8SLlZDcD/WCEUrtsSUn39E9w9NEHfTD6NHeOKvu3EoYFkJzfvNwLxv7KUgm1EYFr8PPt/THbuCxkkXy5qfxLvz4NQ2EI7+f8G9sbz15tb1W+B9hOH0AnFeBGFxv7KEYfEKdE7nPvB/UB71mlWH8cML+Ne4OJ5fKoW5qmkSCjzCttEdMeOEDRo2KA+zl/dw9iwwZMMejKx/HYu694P/jRfCabwwipq3wnzhi6ORdOtnNWKqNEc16zjc/TsEkdYj8dv+8dLIWtxWVzTxqocAjZO4ZpjR/PL43nIE7GYcwfPnybAoWhif9tuEdeOr6/fRy9LnuoDm8ZOCX+FWdR8GhK2VpvpIx5P9KJQUAq442pvRyLBCCJuJplcxR2NZMYCm3JgDp6/M3xqGtSslntA6tbuBCZfXwMksCj496+Ja97tY1u/1PMKI1c7oFuSGE/6ub4zuKhCGeY7t8GjITXj1Tjv3ULzYPDTLCbOfzuXIcK4fTdygoQmI3xGz2/ZAaEMf/DSrA2wSz2G9+wj8EFQTP6jOhfeEvuroVRhT/HzxeT1j3Nk7E95/fY4FM23g07kD7vQ+DK+BpWAcfwfrPH9HE/ex0jSH9MOw8kL39xoH3rhDJdorcEUoTyf801V9oWxoLcL6ZlXgfYVhzUBcsmRJiOWQc0RY7SFLGBavTHv622J1sCeaqkajwv17ovO0D7BI9QXwcGsfOC+tBb8DM1PnL4ULP3PxqouNqjCrHS7EkB1y9hM0baya65gYiGH156Daz6GYape9MCzOGeY0iax2C8NdLjfCsPIklDY4iz8Tj7/shmHxInJNfztsLr0OOxc3hYnWPHh1S6V3Yaj+ndgXuy6pDt/j82GrmjsolqVtt434Txh/ruo0D7/92BUZP8ZguMcDa06B7Ai8PP417L5MxMKzP6GNqq9pXlR3kL4X+iBp4iXM66J1YYp7WNy5Nc7YboDnpOaqu6av955eGFb/7KLD3nTDsLi238CyCKh2MMPfZ6d+XDb/CrzPMCyqnjt3Do8ePZLuWtrZ2ckyNUKz9WQIw8qRqqC6QWk6W9rgmfY2rbpACgRjfMNZ+GjjQYwQpl2+OdKmfBDIx/csom0+Qc8BLRA8vHfqA3HZGRlmGM6/nTg3a6ZLYVh8kG7t0HZY8WQ0Vm8bpRodUo70PBuufChUMwzb/doRh7YN0phvn4Aw/wFwmRePMeu2YUh94zeoXkXfwu7vB2HpY3f8vrpLrszVz8324LYooE8C6Z2TNMNwe+FuU/9qK9AsUPksgfZHnCbh/vWPOPRPFIyL1UCXMd74tl9V6W5PRtMk/IdWwJqiW7BPuFhWf5IjQ3A2qgEafngeU1r2xIsRt9LcSdInU5Y1bwTeZxhWT41QKBRISkqS3miR3kN1uSmR62E4ow6qGYalq2HhrQ5bo2xSn4JVV8oIlui1JBju9m+G4avLHODia4OxPrPRuXI0Tqz0wrKAU2jscUsKAgzDuXlocFuigK6EYfEhvPmuQ/FHwW/w68ZBqKzxNPiO0dWxpuQfaS4+90/6BJ5Gftgzv4GqIeNwdIkLxqwtiskBG9BHmI+Y0SdFnJNc4zC+UE0H4ZFAAQq8m0B80FC0mFkBG0/MhPpdSGnD8C4Mr+2Byqq7mxnvJQ7/hq7GyC98UXfJeXzrkHEYjljXDQ4Li8Hz+Bp0Kqrc4jnvtui3siKmTimIRR7RmBGyCT146+fdGtVA1npfYVh7jvC1a9fSfagut5sh18Ow+DCAOGdpU+mANFem2lMS0rt61a6c5siwQrplZIcbn12C77APpEW1gzfDcG4fHoa3PXEqzqHD5WDfvjRMhJFY789b4GSTo9g0sQLEk1j/alvgfGo7+tqIr/7bhqGOXmi49kimc4ZTj1WNOcPiz9JOk9B+OO+1ffzDAHztMgORrVZg5Xy7N54Cl/pW79sYF7QGPYTXqcWe94Bz71AMEt4WMUh456g4orx5fFcsvNYRP/p/h2ZC2dWflLhTmDFkA5rPWyxcYIrTj4TRY7++6OFdE+tOf4e6hncIsMYUyJLAwzMB+KdIN9hLD7oF4aAw8urcqLhwXnqEv7ZdQmnntqhotBvDbSfD3D0Inv1KCSO6qrsz0yywQDVl8Mi0enC/OhqbNg2TLnLF/r79r5bo9Zk1TuzYjZqdPpNecyhud+lnTfBgwG0sEl5uk9HAk7jchoGt4PmgB5at/1b1isU4bBtlC/c/E1B7QNoL5yxVlgsZnMD7CMMZPSyX0VsmcrNRZAjDgDgfuP3Uxxi8YTcmNC4onYx3Te+DiVuqw0v1BRBzfBwcBl+Eq2oZsVLSl0BoG/R1Vr70Je00iQRhrlN1bLTZLM2VFG8TpcTvxUi7kbBwD1OODAe6wVb4kll6+he0F+Yqi+9endJvJE4Ir5IRX42V/qvVbmPSNT84GqV9oj43kbkt/RFIitwhBM+JOPisCMyTY1DoQ833ckbB/ys7zD5mBkuLAvigSlOUiriAFstzHoYrX5+N9p23wM7nIr7TeBG3+oEXv3tpR3KNFOUwNvXWqvA0+uq+cFt4DUaFCyA+tih6zt8Cj8/Eky8gvqKwl++jNxpBeVKsits7xmPgtAN4alYYBZNiEWfyCb5ZtTndaRT605IsKQXkE1D3y72Vl+GkMJ1IHNyZd3MMAg+7o7zUlzeg8cIbUmh9EPwN3L76HY8KWcFcYYqag9sj0ete6p0XReIN/DykD7xDklC4sDFexn+A7nPF/muE38d3h8e+GBSvXAEmD8ORVPFLLFe9JzijMCxdfAvbXDv6Cyw+9FB6D7FJwlMkGlfEJzXiEBbTA7/tdkcNvmtYvgMkH2w5vTBcrFjWXsgnvpc4u3+BTvyDGqdOncrwYTl1IM7udrPaFLKEYfHK9M9pvTHutwiYWFnBOKkQ2g1sgBCfaI0/upEgnYT7TzqA2MKFpSvml/HFhS+BTaknce05w4l3V6KfyyLcKdEYdWye47GiKizu7kDZscowrH6Lhd+dgrAS3nVlYtkGPRqeQsDjb9INwymJuzG66dc4lPgBSjacir1resIiq3JcLl8LvHhyDzEmZdJ93V5mv3t3lATsHT8E/07wk16X9i4f5R+REd6/8k5/rVH5OsMXZiWk1xDyQwEKZC6giItGjGkR6bWKYt+LSSwKK9UD43HR0TAt8voVn+LrDR+HP4Fx8QooWijtW2nUexFfDXr/SQKKaPVfZb+OQYF3ev2ncr+vLJT9Wjw3X79qjJq1irN5KZCpgGYYjhNe7/nw4cNsiZUpUwblypXL1jriX7mrVKlShg/Lve332dqZ1sKyhOG3de60ZVB1VmOrNO8HzrhSyuWTM/xiyN5JPWcBIif0XJcCrwWSHgdizb66GKR6OIY2FKBA/hTQfkVj/qwla6XvApphOLM/kazv9VSXX9YwnF+QWA8KUIACFKBAbggocAJLh55Dm5UjUS83NshtUEAGAXUYzs6mxWkUtra22VlFZ5ZlGNaZpmBBKEABClCAAhSgwPsXEOfwRkREZKsg4h/HyO7UiGztQMaFGYZlxOWmKUABClCAAhSgAAV0W4BhWLfbh6WjAAUoQAEKUIACFJBRgGFYRlxumgIUoAAFKEABClBAtwUYhnW7fVg6ClCAAhSgAAUoQAEZBRiGZcTlpilAAQpQgAIUoAAFdFuAYVi324elowAFKEABClCAAhSQUYBhWEZcbpoCFKAABShAAQpQQLcFGIZ1u31YOgpQgAIUoAAFKEABGQUYhmXE5aYpQAEKUIACFKAABXRbgGFYt9uHpaMABShAAQpQgAIUkFGAYVhGXG6aAhSgAAUoQAEKUEC3BRiGdbt9WDoKUIACFKAABShAARkFGIZlxOWmKUABClCAAhSgAAV0W4BhWLfbh6WjAAUoQAEKUIACFJBRgGFYRlxumgIUoAAFKEABClBAtwUYhnW7fVg6ClCAAhSgAAUoQAEZBRiGZcTlpilAAQpQgAIUoAAFdFuAYVi324elowAFKEABClCAAhSQUYBhWEZcbpoCFKAABShAAQpQQLcFGIZ1u31YOgpQgL34LDMAAACaSURBVAIUoAAFKEABGQUYhmXE5aYpQAEKUIACFKAABXRbgGFYt9uHpaMABShAAQpQgAIUkFGAYVhGXG6aAhSgAAUoQAEKUEC3BRiGdbt9WDoKUIACFKAABShAARkFGIZlxOWmKUABClCAAhSgAAV0W4BhWLfbh6WjAAUoQAEKUIACFJBRgGFYRlxumgIUoAAFKEABClBAtwX+D4/nW2ut3arDAAAAAElFTkSuQmCC)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "WSI47-j0DBur"
+   },
+   "source": [
+    "6. Click the two overlaid squares ![image.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAEEAAAA/CAYAAAC/36X0AAABf0lEQVR4Xu2aPYqFMBhFvwcWKlhoqaWFG9DNuwVXYGGprYUgdjMRMTgyPHRiRpk5AeEJ5u+8e6+R5PWhivzz8gKCCBCUC4AAhCUMUQIQUIJeGGAH7IAdsMP2S4FMuDoT+r6Xqqoe8TlWFIWEYXhoLJcqAQiKORC+geC6rvi+f0iSpg+N4yjTNOlmHmOHNE1lvn6jNE0j87UWICgSQLANYQ6/tTiOI0EQ6Pt9MP5ZO5RlqScdRZHkeQ4EIKAEEeygkgAIQFheCCgBCCiBxRIrRqUBIABhiQKUAASUoF+L2AE7YAfssN03IBPIhPOZEMexJEmyFZK1323bStd1un2rmy9n7GBtxgcaBoKCBIQ7IQzDIHVdHxCr/UeyLPuyRfiux9MnVd5lgv2p2ekBCIorEEwheJ4n81rgjjIfBbpqDWKkhDsmv/a53/gxGQsQTO1gQt+07q1K2B7XMZ2ISf39USGTtk7bwaSzp9YFwk8y4an/psm4UAJKWPSDEoCwKOET7bDhYssL2DsAAAAASUVORK5CYII=) to copy the API key; then run the code below and paste it into the input text box and press Enter."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "uNA8U5gvDBus"
+   },
+   "outputs": [],
+   "source": [
+    "os.environ['ACTIVELOOP_TOKEN'] = getpass(\"DeepLake API key: \")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "8CmfEgseDBus"
+   },
+   "source": [
+    "Finally, pick a name for your dataset. It doesn't matter what this is, but keep in mind that it will be shared with the students."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "dosq4FolDBus"
+   },
+   "outputs": [],
+   "source": [
+    "dataset_name = input(\"Enter a name for your dataset: \")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "m0zs864gDBut"
+   },
+   "source": [
+    "## Processing The Document(s)\n",
+    "\n",
+    "In this part, you will upload the documents you want the students / model to reference; the embeddings will be created from those documents.\n",
+    "\n",
+    "**Note: The embeddings of all the documents you share will be publicly available. Do not use this for any documents you want to keep private.**"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "eOEJ7rnlDBut"
+   },
+   "source": [
+    "First, upload your documents to Google Colab. To do this:\n",
+    "1. Click on the 'file' icon ![image1_3.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAFQAAABUCAYAAAAcaxDBAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAALEoAACxKAXd6dE0AAAIKSURBVHhe7do/kmFRFMfxM7MAGyCRKZEI60CA0CJkEjKJHZApAevwJxJhAWyABfTM6T7JdDZ1v6/rvfL7VHV1ndfJ7W9f+ql3f338ZYL5Hd8FoqAwBYUpKExBYQoKU1CYgsIUFKagMAWFKShMQWEKClNQmILCFBSmoDAFhSkoTEFhCgpTUJiCwrCDDs/n01arlW23W3s8HnH1/83nc+v1ejEVDxLUYw4GA7ter3ElTZGjIi/56XSKxXTj8fhzpxdR8g713dloNGJiFXGnJgc9HA42HA5j4k0mE6vX6zExyuWyVSqVmFi5D5qVVquVyR/rbW+bjsfj5z/Sy+USVxhvfR/6er1sNpvFxHj7G3vfqff7PaZ0mbyHNptNa7fbMeWLr/d0OsX0Zb1ec+v1oCn2+/1HtVr952uxWMRP88fX9n29/jtQ9FkepqAwBYUpKExBYQoKU1CYgsIUFKagMAWFKShMQWEKClNQmILCFBSmoDAFhSkoTEFhmTxGzvLsUCp/Bv/9/Cr5GDnXp+9+yvl8tlKpFFOa5Je8L6TT6cRUPL52KqZL3qHOd2m/37fb7RZXiqFWq9lms8lfUOdRl8ul7Xa7pDP2P8Hf47vdro1GIzSmw4LKF902wRQUpqAwBYUpKExBYQoKU1CYgsIUFKagMAWFKShMQWEKClNQmILCFBSmoDAFhSkoyuwPHzLZbD4ZgJMAAAAASUVORK5CYII=) at the bottom of the sidebar to the left of these instructions.\n",
+    "2. Click on the 'upload file' icon ![image2_1.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAFcAAABaCAYAAADNVsqyAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAALEoAACxKAXd6dE0AAAKKSURBVHhe7dw7bsJAFIVhJx3UtLARaNkILIKakkXAGhAlNSyCmj3QJjnRjOQgY173eO5kzi+h2Fgok08Tg8zAx9dPlaL0GX4qQsIlJlxiwiUmXGLCJSZcYsIlJlxiwiUmXGLCJSZcYsIlJlxiwiUmXGLCJUZ9m+d4PFb7/b46n8/hHm7j8biaz+dhL30U3MvlUq1Wq85Q63kCppwWUsEi/Les1+uwlzZz3MPhkAw25gXYHBd/mIc8AJvjnk6nsJW+1MDmT2iz2Sxs/W2z2YQt2279vnqpnuSKeJ2bagYXgYtSABeDi7oGLgoXdQlcHC7qCrhIXNQFcLG4iA1cNC5iAhePi1jAwg0xgIVbyxpYuFcB2CrhEhMusewvOb4Te6yaucSES0y4xIRLTLjEssLFmgjcmFeyLMsGF6h4iYQb80qWZVngRth6OQC7x22CjXkHdo3bBhvzDOwW9xHYGIC3223Y85NL3GdgY7vd7vdxnnKH+wpsDI/zBOwK9x3YmCdgN7gWsDEvwC5wLWFjHoCT496D7fV6Yau5tuOpgZPjtq1EB9xisQh7zeF4G3DKle7JcbHiGyu/r4uwo9Eo3NMcjt8CTv2xKRfn3Gvg4XBYLZfLu7CxJmAPn0dzgYsiMGABNRgMwpHHqgN7gEVucBFAANTv98M9zwVgzHgPsMgVLnoVNvbsjGfmDvc/JVxiwiUmXGLCJSZcYlrl2JDVWDVziQmXWGenhZxye1rAhZecsxy/Oe50Og1beWY5fnPcyWSS7ezFuDF+qyhPaLhsmBswxotxW0b5RrwYvmoQS41Sf89YW0DFqcByxsaouKWn17nEhEtMuMSES0y4xIRLTLjEhEtMuMSES6uqvgFS+TQXb05HUQAAAABJRU5ErkJggg==) on the left of the Files toolbar.\n",
+    "3. Select all of the files you want to upload, then click 'Open'.\n",
+    "4. A warning should pop up. Click 'OK' to continue.\n",
+    "5. Wait until the spinning circle in the bottom of the 'Files' section disappears. This means that all of the files have been uploaded."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "hcRqREcvIeY7"
+   },
+   "source": [
+    "### Adding YouTube Videos / Websites\n",
+    "If you have any websites or YouTube videos which also contain content which you want to put into your data lake, paste those links one at a time into the text box below, pressing 'Enter' after each one. Once you have entered all the links, press 'Enter' without typing anything to finish execution of the code cell.\n",
+    "\n",
+    "If you have no URLs to add, just click on the box and press 'Enter' without typing anything."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "19tljTkqIiz9"
+   },
+   "outputs": [],
+   "source": [
+    "url_list = []\n",
+    "while (url := input(\"Enter a YouTube / website link: \")): url_list.append(url)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "qAVRPqDsDBuu"
+   },
+   "source": [
+    "## Embedding & Database Creation"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "_N4erDDkDBuu"
+   },
+   "source": [
+    "Finally, run the following code."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "G2DNpm5dDqJ8"
+   },
+   "outputs": [],
+   "source": [
+    "# Functions to extract the text from YouTube videos / websites\n",
+    "def save_text(text, text_name = None):\n",
+    "  if not text_name: text_name = text[:20]\n",
+    "  text_path = os.path.join(\"/content\",text_name+\".txt\")\n",
+    "  with open(text_path, \"x\") as f:\n",
+    "    f.write(text)\n",
+    "  # Return the location at which the transcript is saved\n",
+    "  return text_path\n",
+    "\n",
+    "def save_youtube_transcript(url, save_dir = \"sample_data\"):\n",
+    "  # Transcribe the videos to text and save to file in /content\n",
+    "  # save_dir: directory to save audio files\n",
+    "  youtube_loader = GenericLoader(YoutubeAudioLoader([url], save_dir),\n",
+    "                                 OpenAIWhisperParser())\n",
+    "  youtube_docs = youtube_loader.load()\n",
+    "  # Combine doc\n",
+    "  combined_docs = [doc.page_content for doc in youtube_docs]\n",
+    "  text = \" \".join(combined_docs)\n",
+    "  # Save text to file\n",
+    "  video_path = youtube_docs[0].metadata[\"source\"]\n",
+    "  youtube_name = os.path.splitext(os.path.basename(video_path))[0]\n",
+    "  return save_text(text, youtube_name)\n",
+    "\n",
+    "# Multiple ways of extracting web content\n",
+    "\n",
+    "def website_webbase(url):\n",
+    "  website_loader = WebBaseLoader(url)\n",
+    "  website_data = website_loader.load()\n",
+    "  # Combine doc\n",
+    "  combined_docs = [doc.page_content for doc in website_data]\n",
+    "  text = \" \".join(combined_docs)\n",
+    "  return text\n",
+    "\n",
+    "def website_trafilatura(url):\n",
+    "  downloaded = trafilatura.fetch_url(url)\n",
+    "  return trafilatura.extract(downloaded)\n",
+    "\n",
+    "def website_justext(url):\n",
+    "  response = requests.get(url)\n",
+    "  paragraphs = justext.justext(response.content, justext.get_stoplist(\"English\"))\n",
+    "  content = [paragraph.text for paragraph in paragraphs \\\n",
+    "            if not paragraph.is_boilerplate]\n",
+    "  text = \" \".join(content)\n",
+    "  return text\n",
+    "\n",
+    "def get_website_youtube_text_file(urls):\n",
+    "  for url in urls:\n",
+    "    # This is a bit of a hacky way to determine whether it's youtube video\n",
+    "    if \"youtube.com\" in url or \"youtu.be\" in url: save_youtube_transcript(url)\n",
+    "    else: save_text(website_webbase(url))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "H3gtJjSfoK5C"
+   },
+   "outputs": [],
+   "source": [
+    "get_website_youtube_text_file(url_list)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "gFNbFdEKDBuu"
+   },
+   "outputs": [],
+   "source": [
+    "### Embedding Creation ###\n",
+    "\n",
+    "# See https://github.com/hwchase17/langchain/discussions/3786\n",
+    "# for discussion of which splitter to use\n",
+    "text_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=0)\n",
+    "\n",
+    "all_document_segments = []\n",
+    "print(\"The following files have been uploaded:\")\n",
+    "for filename in os.listdir():\n",
+    "  if filename not in [\".config\", \"sample_data\"]:\n",
+    "    print(\"\\t\"+filename)\n",
+    "    loader = UnstructuredFileLoader(filename)\n",
+    "    documents = loader.load()\n",
+    "    document_segments = text_splitter.split_documents(documents)\n",
+    "    all_document_segments.extend(document_segments)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "cBcOSf2rDBuv"
+   },
+   "source": [
+    "Make sure that all of your documents are shown in the output from the previous code cell, then continue execution."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "HH8WMgWSDBuw"
+   },
+   "outputs": [],
+   "source": [
+    "model_name = 'text-embedding-ada-002'\n",
+    "model_embedding_dimension = 1536\n",
+    "\n",
+    "embeddings = OpenAIEmbeddings(\n",
+    "    model=model_name\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "HXJYOjYKDBuw"
+   },
+   "outputs": [],
+   "source": [
+    "### Dataset Creation ###\n",
+    "dataset_path = f\"hub://{DEEPLAKE_USERNAME}/{dataset_name}\"\n",
+    "db = DeepLake.from_documents(all_document_segments, dataset_path=dataset_path,\n",
+    "                             embedding=embeddings, public=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "01UMqQcHDBux"
+   },
+   "source": [
+    "### Sharing With Students"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "l3TMy84LDBux"
+   },
+   "outputs": [],
+   "source": [
+    "display(Markdown(f'''To let students access the repository, give them the following URL:\n",
+    "\n",
+    "`{dataset_path}`'''))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "iUeTuV-WDBuy"
+   },
+   "source": [
+    "Distribute the URL above to students. They will copy and paste it into the LLM learning application, which then allows their models to use all of the documents you uploaded as reference sources when responding to or creating questions."
+   ]
+  }
+ ],
+ "metadata": {
+  "colab": {
+   "include_colab_link": true,
+   "provenance": []
+  },
+  "kernelspec": {
+   "display_name": "Python 3",
+   "name": "python3"
+  },
+  "language_info": {
+   "name": "python"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}

prompt_with_context.ipynb

@@ -0,0 +1,962 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "colab_type": "text",
+        "id": "view-in-github"
+      },
+      "source": [
+        "<a href=\"https://colab.research.google.com/github/vanderbilt-data-science/lo-achievement/blob/main/prompt_with_context.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "S4MkldrwPA_S"
+      },
+      "source": [
+        "# LLMs for Self-Study\n",
+        "> A prompt and code template for better understanding texts\n",
+        "\n",
+        "This notebook provides a guide for using LLMs for self-study programmatically. A number of prompt templates are provided to assist with generating great assessments for self-study, and code is additionally provided for fast usage. This notebook is best leveraged for a set of documents (text or PDF preferred) **to be uploaded** for interaction with the model.\n",
+        "\n",
+        "This version of the notebook is best suited for those who prefer to use files from their local drive as context rather than copy and pasting directly into the notebook to be used as context for the model. If you prefer to copy and paste text, you should direct yourself to the [prompt_with_context](https://colab.research.google.com/github/vanderbilt-data-science/lo-achievement/blob/main/prompt_with_context.ipynb) notebook."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "6Gf-_aZVPA_S"
+      },
+      "source": [
+        "# Code Setup\n",
+        "Run the following cells to setup the rest of the environment for prompting. In the following section, we set up the computational environment with imported code, setup your API key access to OpenAI, and loading access to your language model. Note that the following cells may take a long time to run."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "6040J5eXPA_T"
+      },
+      "source": [
+        "## Library installation and loading\n",
+        "The following `pip install` code should be run if you're using Google Colab, or otherwise do not have a computational environment (e.g., _venv_, _conda virtual environment_, _Docker, Singularity, or other container_) with these packages installed."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 1,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "7_XCtEMbPA_T",
+        "outputId": "aa36bcf0-1d11-4947-f96b-d28c67d4e431"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "\u001b[33mDEPRECATION: pyodbc 4.0.0-unsupported has a non-standard version number. pip 23.3 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pyodbc or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063\u001b[0m\u001b[33m\n",
+            "\u001b[0m"
+          ]
+        }
+      ],
+      "source": [
+        "# run this code if you're using Google Colab or don't have these packages installed in your computing environment\n",
+        "! pip install -q langchain openai gradio numpy tiktoken"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 2,
+      "metadata": {
+        "id": "6fdfMar8PA_U"
+      },
+      "outputs": [],
+      "source": [
+        "# import required libraries\n",
+        "import numpy as np\n",
+        "import getpass\n",
+        "import os\n",
+        "from langchain.chat_models import ChatOpenAI\n",
+        "from langchain.chains import RetrievalQA\n",
+        "from langchain.schema import SystemMessage, HumanMessage, AIMessage"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "5iI75yjaPA_V"
+      },
+      "source": [
+        "## API and model setup\n",
+        "\n",
+        "Use these cells to load the API keys required for this notebook and create a basic OpenAI LLM model. The code below uses the variable you created above when you input your API Key."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 3,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "8IgmjGEFPA_V",
+        "outputId": "5ac82647-2f93-4bd3-9284-e18dd04b1c81"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "··········\n"
+          ]
+        }
+      ],
+      "source": [
+        "# Set up OpenAI API Key\n",
+        "openai_api_key = getpass.getpass()\n",
+        "os.environ[\"OPENAI_API_KEY\"] = openai_api_key\n",
+        "\n",
+        "llm = ChatOpenAI(model='gpt-3.5-turbo-16k')\n",
+        "messages = [\n",
+        "    SystemMessage(content=\"You are a world-class tutor helping students to perform better on oral and written exams though interactive experiences.\"),\n",
+        "    HumanMessage(content=\"\")\n",
+        "]\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "Zlx1_Y8rYGn3"
+      },
+      "source": [
+        "# Add your context and assign the prefix to your query.\n",
+        "The query assigned here serves as an example."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 4,
+      "metadata": {
+        "id": "g_CAvacjEuCd"
+      },
+      "outputs": [],
+      "source": [
+        "context = \"\"\" Two roads diverged in a yellow wood,\n",
+        "And sorry I could not travel both\n",
+        "And be one traveler, long I stood\n",
+        "And looked down one as far as I could\n",
+        "To where it bent in the undergrowth;\n",
+        "Then took the other, as just as fair,\n",
+        "And having perhaps the better claim,\n",
+        "Because it was grassy and wanted wear;\n",
+        "Though as for that the passing there\n",
+        "Had worn them really about the same,\n",
+        "And both that morning equally lay\n",
+        "In leaves no step had trodden black.\n",
+        "Oh, I kept the first for another day!\n",
+        "Yet knowing how way leads on to way,\n",
+        "I doubted if I should ever come back.\n",
+        "I shall be telling this with a sigh\n",
+        "Somewhere ages and ages hence:\n",
+        "Two roads diverged in a wood, and I—\n",
+        "I took the one less traveled by,\n",
+        "And that has made all the difference.\n",
+        "—-Robert Frost—-\n",
+        "Education Place: http://www.eduplace.com \"\"\"\n",
+        "\n",
+        "# Query prefix\n",
+        "query_prefix = \"The following text should be used as the basis for the instructions which follow: \" + context + '\\n'\n",
+        "\n",
+        "# Query\n",
+        "query = \"\"\"Please design a 5 question quiz about Robert Frost's \"Road Not Taken\" which reflects the learning objectives:\n",
+        "1. Identify the key elements of the poem: narrator, setting, and underlying message.\n",
+        "2. Understand the literary devices used in poetry and their purposes. The questions should be multiple choice.\n",
+        "Provide one question at a time, and wait for my response before providing me with feedback.\n",
+        "Again, while the quiz asks for 5 questions, you should\n",
+        "only provide ONE question in you initial response. Do not include the answer in your response.\n",
+        "If I get an answer wrong, provide me with an explanation of why it was incorrect, and then give me additional\n",
+        "chances to respond until I get the correct choice. Explain why the correct choice is right. \"\"\""
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "hLMJRXc8PA_W"
+      },
+      "source": [
+        "# A guide to prompting for self-study\n",
+        "In this section, we provide a number of different approaches for using AI to help you assess and explain the knowledge of your document. Start by interacting with the model and then try out the rest of the prompts!"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "jNBwByBFaJVP"
+      },
+      "source": [
+        "## Interact with the model\n",
+        "\n",
+        "Now that your vector store is created, you can begin interacting with the model! Below, we have a comprehensive list of examples using different question types, but feel free to use this code block to experiment with the model.\n",
+        "\n",
+        "Input your prompt into the empty string in the code cell. See example below:\n",
+        "\n",
+        "\n",
+        "\n",
+        "```\n",
+        "query = 'Your Prompt Here'\n",
+        "```\n",
+        "\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 5,
+      "metadata": {
+        "id": "3fPd2b3yUZic"
+      },
+      "outputs": [],
+      "source": [
+        "def prompt_define(query, prefix, context):\n",
+        "  prompt = (query + prefix)\n",
+        "  prompt = prompt + context\n",
+        "  return prompt\n",
+        "\n",
+        "def get_result(prompt):\n",
+        "  messages[1] = HumanMessage(content=prompt)\n",
+        "  result = llm(messages)\n",
+        "  str_result = str(result)\n",
+        "  import re\n",
+        "  str_result = str_result.replace(r'\\n', '\\n')\n",
+        "  str_result = str_result.replace(\"content='\", \"\")\n",
+        "  str_result = str_result.replace(\"' additional_kwargs={} example=False\", \"\")\n",
+        "  return str_result"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "cskXB5p5DupT"
+      },
+      "outputs": [],
+      "source": [
+        "query = 'Your Prompt Here'\n",
+        "\n",
+        "prompt = prompt_define(query, query_prefix, context)\n",
+        "\n",
+        "result = get_result(prompt)\n",
+        "\n",
+        "print(result)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "FLy4qDuWclWF"
+      },
+      "source": [
+        "### Our example using query from cell 103\n",
+        "\n",
+        "Run the following code to see a simple example using the prompt defined in an earlier cell (#103)."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 6,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "4QBBlJI-adPf",
+        "outputId": "0d421ad5-74ca-40eb-bd68-afcb47768275"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "Question 1: Who is the narrator of the poem \"Road Not Taken\"?\n",
+            "\n",
+            "a) Robert Frost\n",
+            "b) A traveler\n",
+            "c) The poet\\'s friend\n",
+            "d) The reader\n"
+          ]
+        }
+      ],
+      "source": [
+        "# Experiment with interacting with the model by inputting your own prompts into the empty string below.\n",
+        "def prompt_define(query, prefix, context):\n",
+        "  prompt = (query + prefix)\n",
+        "  prompt = prompt + context\n",
+        "  return prompt\n",
+        "\n",
+        "prompt = prompt_define(query, query_prefix, context)\n",
+        "\n",
+        "def get_result(prompt):\n",
+        "  messages[1] = HumanMessage(content=prompt)\n",
+        "  result = llm(messages)\n",
+        "  str_result = str(result)\n",
+        "  import re\n",
+        "  str_result = str_result.replace(r'\\n', '\\n')\n",
+        "  str_result = str_result.replace(\"content='\", \"\")\n",
+        "  str_result = str_result.replace(\"' additional_kwargs={} example=False\", \"\")\n",
+        "  return str_result\n",
+        "\n",
+        "result = get_result(prompt)\n",
+        "\n",
+        "print(result)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "KYro6H82bENS"
+      },
+      "source": [
+        "## Types of Questions and Prompts\n",
+        "\n",
+        "Below is a comprehensive list of question types and prompt templates designed by our team. There are also example code blocks, where you can see how the model performed with the example and try it for yourself using the prompt template."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "WAUSc7qJPA_X"
+      },
+      "source": [
+        "### Multiple Choice\n",
+        "\n",
+        "Prompt: The following text should be used as the basis for the instructions which follow: {context}. Please design a {number of questions} question quiz about {name or reference to context} which reflects the learning objectives: {list of learning objectives}. The questions should be multiple choice. Provide one question at a time, and wait for my response before providing me with feedback. Again, while the quiz may ask for multiple questions, you should only provide ONE question in you initial response. Do not include the answer in your response. If I get an answer wrong, provide me with an explanation of why it was incorrect,and then give me additional chances to respond until I get the correct choice. Explain why the correct choice is right."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 7,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "nFMRDL4cPA_X",
+        "outputId": "b341fcb8-49df-43e6-9810-7ff3f232949e"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "Question 1: Who is the narrator of the poem \"Road Not Taken\"?\n",
+            "\n",
+            "A) Robert Frost\n",
+            "B) The reader\n",
+            "C) The traveler\n",
+            "D) The poet\n"
+          ]
+        }
+      ],
+      "source": [
+        "# Multiple choice code example\n",
+        "query = \"\"\"Please design a 5 question quiz about Robert Frost's \"Road Not Taken\" which reflects the learning objectives:\n",
+        "1. Identify the key elements of the poem: narrator, setting, and underlying message.\n",
+        "2. Understand the literary devices used in poetry and their purposes. The questions should be multiple choice.\n",
+        "Provide one question at a time, and wait for my response before providing me with feedback.\n",
+        "Again, while the quiz asks for 5 questions, you should\n",
+        "only provide ONE question in you initial response. Do not include the answer in your response\n",
+        "If I get an answer wrong, provide me with an explanation of why it was incorrect, and then give me additional\n",
+        "chances to respond until I get the correct choice. Explain why the correct choice is right. \"\"\"\n",
+        "\n",
+        "def prompt_define(query, prefix, context):\n",
+        "  prompt = (query + prefix)\n",
+        "  prompt = prompt + context\n",
+        "  return prompt\n",
+        "\n",
+        "prompt = prompt_define(query, query_prefix, context)\n",
+        "\n",
+        "def get_result(prompt):\n",
+        "  messages[1] = HumanMessage(content=prompt)\n",
+        "  result = llm(messages)\n",
+        "  str_result = str(result)\n",
+        "  import re\n",
+        "  str_result = str_result.replace(r'\\n', '\\n')\n",
+        "  str_result = str_result.replace(\"content='\", \"\")\n",
+        "  str_result = str_result.replace(\"' additional_kwargs={} example=False\", \"\")\n",
+        "  return str_result\n",
+        "\n",
+        "result = get_result(prompt)\n",
+        "\n",
+        "print(result)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "WkSIU94GPA_Y"
+      },
+      "source": [
+        "### Short Answer\n",
+        "\n",
+        "Prompt: Please design a {number of questions} question quiz about {context} which reflects the learning objectives: {list of learning objectives}. The questions should be short answer. Expect the correct answers to be {anticipated length} long. Provide one question at a time, and wait for my response before providing me with feedback. Again, while the quiz may ask for multiple questions, you should only provide ONE question in you initial response. Do not include the answer in your response. If I get an answer wrong, provide me with an explanation of why it was incorrect,and then give me additional chances to respond until I get the correct choice. Explain why the correct choice is right."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 8,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "enC8ydfEPA_Y",
+        "outputId": "639efa60-3304-4b63-c0e8-a975d2d7dfaf"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "Question 1: Who is the narrator of the poem \"Road Not Taken\"?\n"
+          ]
+        }
+      ],
+      "source": [
+        "# Short answer code example\n",
+        "query = \"\"\" Please design a 5-question quiz about Robert Frost's\n",
+        "\"Road Not Taken\" which reflects the learning objectives:\n",
+        "1. Identify the key elements of the poem: narrator, setting, and underlying message.\n",
+        "2. Understand the literary devices used in poetry and their purposes.\n",
+        "The questions should be short answer. Expect the correct answers to be\n",
+        "1-2 sentences long.\n",
+        "Provide one question at a time, and wait for my response before providing me with feedback.\n",
+        "Again, while the quiz asks for 5 questions, you should\n",
+        "only provide ONE question in you initial response. Do not include the answer in your response\n",
+        "If I get any part of the answer wrong,\n",
+        "provide me with an explanation of why it was incorrect,\n",
+        "and then give me additional chances to respond until I get the correct choice. \"\"\"\n",
+        "\n",
+        "def prompt_define(query, prefix, context):\n",
+        "  prompt = (query + prefix)\n",
+        "  prompt = prompt + context\n",
+        "  return prompt\n",
+        "\n",
+        "prompt = prompt_define(query, query_prefix, context)\n",
+        "\n",
+        "def get_result(prompt):\n",
+        "  messages[1] = HumanMessage(content=prompt)\n",
+        "  result = llm(messages)\n",
+        "  str_result = str(result)\n",
+        "  import re\n",
+        "  str_result = str_result.replace(r'\\n', '\\n')\n",
+        "  str_result = str_result.replace(\"content='\", \"\")\n",
+        "  str_result = str_result.replace(\"' additional_kwargs={} example=False\", \"\")\n",
+        "  return str_result\n",
+        "\n",
+        "result = get_result(prompt)\n",
+        "\n",
+        "print(result)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "Ta4A527kPA_Y"
+      },
+      "source": [
+        "### Fill-in-the-blank\n",
+        "\n",
+        "Prompt: Create a {number of questions} question fill in the blank quiz refrencing {context}. The quiz should reflect the learning objectives: {learning objectives}. The \"blank\" part of the question should appear as \"________\". The answers should reflect what word(s) should go in the blank an accurate statement.\n",
+        "\n",
+        "An example is the follow: \"The author of the book is \"________.\"\n",
+        "\n",
+        "The question should be a statement. Provide one question at a time, and wait for my response before providing me with feedback. Again, while the quiz may ask for multiple questions, you should only provide ONE question in you initial response. Do not include the answer in your response. If I get an answer wrong, provide me with an explanation of why it was incorrect,and then give me additional chances to respond until I get the correct choice. Explain why the correct choice is right."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 9,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "szMOxEoiPA_Z",
+        "outputId": "0c4c823b-6f07-4ba0-cbf3-54b3e28d0d6c"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "Question 1: In \"The Road Not Taken,\" the narrator comes to a point where two roads ____________ in a yellow wood.\n"
+          ]
+        }
+      ],
+      "source": [
+        "# Fill in the blank code example\n",
+        "query = \"\"\" Create a 5 question fill in the blank quiz refrencing Robert Frost's \"The Road Not Taken.\"\n",
+        "The \"blank\" part of the question should appear as \"________\". The answers should reflect what word(s) should go in the blank an accurate statement.\n",
+        "An example is the follow: \"The author of the book is ______.\" The question should be a statement.\n",
+        "The quiz should reflect the learning objectives:\n",
+        "1. Identify the key elements of the poem: narrator, setting, and underlying message.\n",
+        "2. Understand the literary devices used in poetry and their purposes.\n",
+        "Provide one question at a time, and wait for my response before providing me with feedback.\n",
+        "Again, while the quiz asks for 5 questions, you should\n",
+        "only provide ONE question in you initial response. Do not include the answer in your response\n",
+        "If I answer incorrectly, please explain why my answer is incorrect. \"\"\"\n",
+        "\n",
+        "def prompt_define(query, prefix, context):\n",
+        "  prompt = (query + prefix)\n",
+        "  prompt = prompt + context\n",
+        "  return prompt\n",
+        "\n",
+        "prompt = prompt_define(query, query_prefix, context)\n",
+        "\n",
+        "def get_result(prompt):\n",
+        "  messages[1] = HumanMessage(content=prompt)\n",
+        "  result = llm(messages)\n",
+        "  str_result = str(result)\n",
+        "  import re\n",
+        "  str_result = str_result.replace(r'\\n', '\\n')\n",
+        "  str_result = str_result.replace(\"content='\", \"\")\n",
+        "  str_result = str_result.replace(\"' additional_kwargs={} example=False\", \"\")\n",
+        "  return str_result\n",
+        "\n",
+        "result = get_result(prompt)\n",
+        "\n",
+        "print(result)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "yEwzAB28PA_Z"
+      },
+      "source": [
+        "### Sequencing\n",
+        "\n",
+        "Prompt: Please develop a {number of questions} question questionnaire that will ask me to recall the steps involved in the following learning objectives in regard to {context}: {learning objectives}. Provide one question at a time, and wait for my response before providing me with feedback. Again, while the quiz may ask for multiple questions, you should only provide ONE question in you initial response. Do not include the answer in your response. If I get an answer wrong, provide me with an explanation of why it was incorrect, and then give me additional chances to respond until I get the correct choice. After I respond, explain their sequence to me."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 10,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "3YRyhhWtPA_Z",
+        "outputId": "7e26ed3b-99c7-4577-c808-fb85c3e11b4c"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "Question 1: Who is the narrator of the poem?\n"
+          ]
+        }
+      ],
+      "source": [
+        "# Sequence example\n",
+        "query = \"\"\" Please develop a 5 question questionnaire that will ask me to recall the steps involved in the following learning objectives in regard to Robert Frost's \"The Road Not Taken\":\n",
+        "1. Identify the key elements of the poem: narrator, setting, and underlying message.\n",
+        "2. Understand the literary devices used in poetry and their purposes.\n",
+        "Provide one question at a time, and wait for my response before providing me with feedback.\n",
+        "Again, while the quiz asks for 5 questions, you should\n",
+        "only provide ONE question in you initial response. Do not include the answer in your response.\n",
+        "After I respond, explain their sequence to me.\"\"\"\n",
+        "\n",
+        "def prompt_define(query, prefix, context):\n",
+        "  prompt = (query + prefix)\n",
+        "  prompt = prompt + context\n",
+        "  return prompt\n",
+        "\n",
+        "prompt = prompt_define(query, query_prefix, context)\n",
+        "\n",
+        "def get_result(prompt):\n",
+        "  messages[1] = HumanMessage(content=prompt)\n",
+        "  result = llm(messages)\n",
+        "  str_result = str(result)\n",
+        "  import re\n",
+        "  str_result = str_result.replace(r'\\n', '\\n')\n",
+        "  str_result = str_result.replace(\"content='\", \"\")\n",
+        "  str_result = str_result.replace(\"' additional_kwargs={} example=False\", \"\")\n",
+        "  return str_result\n",
+        "\n",
+        "result = get_result(prompt)\n",
+        "\n",
+        "print(result)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "0DGBiJofPA_Z"
+      },
+      "source": [
+        "### Relationships/drawing connections\n",
+        "\n",
+        "Prompt: Please design a {number of questions} question quiz that asks me to explain the relationships that exist within the following learning objectives, referencing {context}: {learning objectives}. Provide one question at a time, and wait for my response before providing me with feedback. Again, while the quiz may ask for multiple questions, you should only provide ONE question in you initial response. Do not include the answer in your response. If I get an answer wrong, provide me with an explanation of why it was incorrect,and then give me additional chances to respond until I get the correct choice. Explain why the correct choice is right."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 11,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "Mw3RBpsnPA_a",
+        "outputId": "3f721340-7fef-4617-cc2c-2420779ecb2d"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "Question 1: Identify the key elements of the poem: narrator, setting, and underlying message.\n"
+          ]
+        }
+      ],
+      "source": [
+        "# Relationships example\n",
+        "query = \"\"\" Please design a 5 question quiz that asks me to explain the relationships that exist within the following learning objectives, referencing Robert Frost's \"The Road Not Taken\":\n",
+        "1. Identify the key elements of the poem: narrator, setting, and underlying message.\n",
+        "2. Understand the literary devices used in poetry and their purposes.\n",
+        "Provide one question at a time, and wait for my response before providing me with feedback.\n",
+        "Again, while the quiz asks for 5 questions, you should\n",
+        "only provide ONE question in you initial response. Do not include the answer in your response.\"\"\"\n",
+        "\n",
+        "def prompt_define(query, prefix, context):\n",
+        "  prompt = (query + prefix)\n",
+        "  prompt = prompt + context\n",
+        "  return prompt\n",
+        "\n",
+        "prompt = prompt_define(query, query_prefix, context)\n",
+        "\n",
+        "def get_result(prompt):\n",
+        "  messages[1] = HumanMessage(content=prompt)\n",
+        "  result = llm(messages)\n",
+        "  str_result = str(result)\n",
+        "  import re\n",
+        "  str_result = str_result.replace(r'\\n', '\\n')\n",
+        "  str_result = str_result.replace(\"content='\", \"\")\n",
+        "  str_result = str_result.replace(\"' additional_kwargs={} example=False\", \"\")\n",
+        "  return str_result\n",
+        "\n",
+        "result = get_result(prompt)\n",
+        "\n",
+        "print(result)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "4YO3wCTwPA_a"
+      },
+      "source": [
+        "### Concepts and Definitions\n",
+        "\n",
+        "Prompt: Design a {number of questions} question quiz that asks me about definitions related to the following learning objectives: {learning objectives} - based on {context}\".\n",
+        "Provide one question at a time, and wait for my response before providing me with feedback. Again, while the quiz may ask for multiple questions, you should only provide ONE question in you initial response. Do not include the answer in your response. If I get an answer wrong, provide me with an explanation of why it was incorrect,and then give me additional chances to respond until I get the correct choice. Explain why the correct choice is right.\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 12,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "TbvJPhkmPA_a",
+        "outputId": "f174e720-a31a-4476-8701-36ed9f637770"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "Question 1: Who is the narrator of the poem \"The Road Not Taken\"?\n",
+            "\n",
+            "Please provide your response.\n"
+          ]
+        }
+      ],
+      "source": [
+        "# Concepts and definitions example\n",
+        "query = \"\"\" Design a 5 question quiz that asks me about definitions related to the following learning objectives:\n",
+        "1. Identify the key elements of the poem: narrator, setting, and underlying message, and\n",
+        "2. Understand the literary devices used in poetry and their purposes - based on Robert Frost's \"The Road Not Taken\".\n",
+        "Provide one question at a time, and wait for my response before providing me with feedback.\n",
+        "Again, while the quiz asks for 5 questions, you should\n",
+        "only provide ONE question in you initial response. Do not include the answer in your response\n",
+        "Once I write out my response, provide me with your own response, highlighting why my answer is correct or incorrect.\"\"\"\n",
+        "\n",
+        "def prompt_define(query, prefix, context):\n",
+        "  prompt = (query + prefix)\n",
+        "  prompt = prompt + context\n",
+        "  return prompt\n",
+        "\n",
+        "prompt = prompt_define(query, query_prefix, context)\n",
+        "\n",
+        "def get_result(prompt):\n",
+        "  messages[1] = HumanMessage(content=prompt)\n",
+        "  result = llm(messages)\n",
+        "  str_result = str(result)\n",
+        "  import re\n",
+        "  str_result = str_result.replace(r'\\n', '\\n')\n",
+        "  str_result = str_result.replace(\"content='\", \"\")\n",
+        "  str_result = str_result.replace(\"' additional_kwargs={} example=False\", \"\")\n",
+        "  return str_result\n",
+        "\n",
+        "result = get_result(prompt)\n",
+        "\n",
+        "print(result)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "vc-llAgfPA_a"
+      },
+      "source": [
+        "### Real Word Examples\n",
+        "\n",
+        "Prompt: Demonstrate how {context} can be applied to solve a real-world problem related to the following learning objectives: {learning objectives}. Ask me questions regarding this theory/concept.\n",
+        "\n",
+        "Provide one question at a time, and wait for my response before providing me with feedback. Again, while the quiz may ask for multiple questions, you should only provide ONE question in you initial response. Do not include the answer in your response. If I get an answer wrong, provide me with an explanation of why it was incorrect,and then give me additional chances to respond until I get the correct choice. Explain why the correct choice is right."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 14,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "sSwRLkR8PA_a",
+        "outputId": "4d91df15-b7b8-4bc9-d1b2-3af7e629be4f"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "Question 1: Who is the narrator of the poem?\n"
+          ]
+        }
+      ],
+      "source": [
+        "# Real word example\n",
+        "query = \"\"\" Demonstrate how Robert Frost’s “The Road Not Taken” can be applied to solve a real-world problem. Ask me questions regarding\n",
+        "this theory/concept and relate them to the following learning objectives:\n",
+        "1. Identify the key elements of the poem: narrator, setting, and underlying message.\n",
+        "2. Understand the literary devices used in poetry and their purposes.\n",
+        "Provide one question at a time, and wait for my response before providing me with feedback.\n",
+        "Again, while the quiz asks for 5 questions, you should\n",
+        "only provide ONE question in you initial response. Do not include the answer in your response\n",
+        "\"\"\"\n",
+        "\n",
+        "def prompt_define(query, prefix, context):\n",
+        "  prompt = (query + prefix)\n",
+        "  prompt = prompt + context\n",
+        "  return prompt\n",
+        "\n",
+        "prompt = prompt_define(query, query_prefix, context)\n",
+        "\n",
+        "def get_result(prompt):\n",
+        "  messages[1] = HumanMessage(content=prompt)\n",
+        "  result = llm(messages)\n",
+        "  str_result = str(result)\n",
+        "  import re\n",
+        "  str_result = str_result.replace(r'\\n', '\\n')\n",
+        "  str_result = str_result.replace(\"content='\", \"\")\n",
+        "  str_result = str_result.replace(\"' additional_kwargs={} example=False\", \"\")\n",
+        "  return str_result\n",
+        "\n",
+        "result = get_result(prompt)\n",
+        "\n",
+        "print(result)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "2y6gHhGKPA_b"
+      },
+      "source": [
+        "### Randomized Question Types\n",
+        "\n",
+        "Prompt: Please generate a high-quality assessment consisting of {number of questions} varying questions, each of different types (open-ended, multiple choice, etc.), to determine if I achieved the following learning objectives in regards to {context}: {learning objectives}.\n",
+        "\n",
+        "Provide one question at a time, and wait for my response before providing me with feedback. Again, while the quiz may ask for multiple questions, you should only provide ONE question in you initial response. Do not include the answer in your response. If I get an answer wrong, provide me with an explanation of why it was incorrect,and then give me additional chances to respond until I get the correct choice. Explain why the correct choice is right."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 15,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "xC4t4WMwPA_b",
+        "outputId": "331b1ef7-ef34-4d21-ddbd-e2ebae1d2fd4"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "Question 1: \n",
+            "Identify the narrator of the poem \"The Road not Taken\" by Robert Frost.\n",
+            "\n",
+            "Question 2: \n",
+            "What is the setting of the poem \"The Road not Taken\" by Robert Frost?\n",
+            "\n",
+            "Question 3: \n",
+            "What is the underlying message or theme of the poem \"The Road not Taken\" by Robert Frost?\n",
+            "\n",
+            "Question 4 (Multiple Choice):\n",
+            "Which literary device is NOT used in the poem \"The Road not Taken\" by Robert Frost?\n",
+            "a) Metaphor\n",
+            "b) Simile\n",
+            "c) Alliteration\n",
+            "d) Personification\n",
+            "\n",
+            "Question 5 (Open-ended):\n",
+            "Provide an example of a literary device used in the poem \"The Road not Taken\" by Robert Frost and explain its purpose.\n"
+          ]
+        }
+      ],
+      "source": [
+        "# Randomized question types\n",
+        "query = \"\"\" Please generate a high-quality assessment consisting of 5 varying questions,\n",
+        "each of different types (open-ended, multiple choice, etc.),\n",
+        "to determine if I achieved the following learning objectives in regards to Robert Frost’s “The Road not Taken\":\n",
+        "1. Identify the key elements of the poem: narrator, setting, and underlying message.\n",
+        "2. Understand the literary devices used in poetry and their purposes. If I answer incorrectly for any of the questions,\n",
+        "please explain why my answer is incorrect.\n",
+        "Provide one question at a time, and wait for my response before providing me with feedback.\n",
+        "Again, while the quiz asks for 5 questions, you should\n",
+        "only provide ONE question in you initial response. Do not include the answer in your response.\n",
+        "\"\"\"\n",
+        "\n",
+        "def prompt_define(query, prefix, context):\n",
+        "  prompt = (query + prefix)\n",
+        "  prompt = prompt + context\n",
+        "  return prompt\n",
+        "\n",
+        "prompt = prompt_define(query, query_prefix, context)\n",
+        "\n",
+        "def get_result(prompt):\n",
+        "  messages[1] = HumanMessage(content=prompt)\n",
+        "  result = llm(messages)\n",
+        "  str_result = str(result)\n",
+        "  import re\n",
+        "  str_result = str_result.replace(r'\\n', '\\n')\n",
+        "  str_result = str_result.replace(\"content='\", \"\")\n",
+        "  str_result = str_result.replace(\"' additional_kwargs={} example=False\", \"\")\n",
+        "  return str_result\n",
+        "\n",
+        "result = get_result(prompt)\n",
+        "\n",
+        "print(result)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "ndo_cdWYPA_b"
+      },
+      "source": [
+        "### Quantiative evaluation the correctness of a student's answer\n",
+        "\n",
+        "Prompt: (A continuation of the previous chat) Please generate the main points of the student’s answer to the previous question, and evaluate on a scale of 1 to 5 how comprehensive the student’s answer was in relation to the learning objectives, and explain why he or she received this rating, including what was missed in his or her answer if the student’s answer wasn’t complete.\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 118,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "WhhNI0FZPA_b",
+        "outputId": "fb037251-2448-44e7-acef-3e6a28da7cb3"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "Question 1: Who is the narrator of the poem \"The Road not Taken\"?\n",
+            "a) Robert Frost\n",
+            "b) The traveler\n",
+            "c) The reader\n",
+            "d) The poet\\'s imagination\n"
+          ]
+        }
+      ],
+      "source": [
+        "# qualitative evaluation\n",
+        "qualitative_query = \"\"\" Please generate the main points of the student’s answer to the previous question,\n",
+        " and evaluate on a scale of 1 to 5 how comprehensive the student’s answer was in relation to the learning objectives,\n",
+        " and explain why he or she received this rating, including what was missed in his or her answer if the student’s answer wasn’t complete.\"\"\"\n",
+        "\n",
+        "# Note that this uses the previous result and query in the context\n",
+        "def prompt_define(query, prefix, context):\n",
+        "  prompt = (query + prefix)\n",
+        "  prompt = prompt + context\n",
+        "  return prompt\n",
+        "\n",
+        "prompt = prompt_define(query, query_prefix, context)\n",
+        "\n",
+        "def get_result(prompt):\n",
+        "  messages[1] = HumanMessage(content=prompt)\n",
+        "  result = llm(messages)\n",
+        "  str_result = str(result)\n",
+        "  import re\n",
+        "  str_result = str_result.replace(r'\\n', '\\n')\n",
+        "  str_result = str_result.replace(\"content='\", \"\")\n",
+        "  str_result = str_result.replace(\"' additional_kwargs={} example=False\", \"\")\n",
+        "  return str_result\n",
+        "\n",
+        "result = get_result(prompt)\n",
+        "\n",
+        "print(result)"
+      ]
+    }
+  ],
+  "metadata": {
+    "colab": {
+      "include_colab_link": true,
+      "provenance": []
+    },
+    "kernelspec": {
+      "display_name": "Python 3.11.5 64-bit",
+      "language": "python",
+      "name": "python3"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.11.5"
+    },
+    "vscode": {
+      "interpreter": {
+        "hash": "b0fa6594d8f4cbf19f97940f81e996739fb7646882a419484c72d19e05852a7e"
+      }
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}