{ "cells": [ { "cell_type": "markdown", "metadata": { "collapsed": true, "id": "OHtb7Wkf1gas" }, "source": [ "\"Open" ] }, { "cell_type": "markdown", "metadata": { "id": "Tq2RwQr71gaz" }, "source": [ "# LLMs for Self-Study\n", "> A prompt and code template for better understanding texts\n", "\n", "This notebook provides a guide for using LLMs for self-study programmatically. A number of prompt templates are provided to assist with generating great assessments for self-study, and code is additionally provided for fast usage. This notebook is best leveraged for a set of documents (text or PDF preferred) **to be uploaded** for interaction with the model.\n", "\n", "This version of the notebook is best suited for those who prefer to use files from their local drive as context rather than copy and pasting directly into the notebook to be used as context for the model. If you prefer to copy and paste text, you should direct yourself to the [prompt_with_context](https://colab.research.google.com/github/vanderbilt-data-science/lo-achievement/blob/main/prompt_with_context.ipynb) notebook." ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "id": "ImAL1pvx1ga0" }, "outputs": [], "source": [ "#libraries for user setup code\n", "from getpass import getpass\n", "from logging import raiseExceptions" ] }, { "cell_type": "markdown", "metadata": { "id": "03JXTFy21ga2" }, "source": [ "## Helper functions\n", "The following functions help to encapsulate the functionality executed below. The `setup_drives` function below assists with setting up the drives for users to upload files." ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "id": "z_l9mjGX1ga3" }, "outputs": [], "source": [ "def setup_drives(upload_set):\n", "\n", " upload_set = upload_set.lower()\n", "\n", " # Colab file upload module\n", " if upload_set == \"local drive\":\n", " from google.colab import files\n", " uploaded = files.upload()\n", " elif upload_set == \"google drive\":\n", " # Mount a Google Drive\n", " from google.colab import drive\n", " drive.mount('/content/drive')\n", " # Raise errors\n", " elif upload_set == '':\n", " raise ValueError(\"You haven't yet defined the upload_settings variable. Go back and read the instructions to make this setting.\")\n", " else:\n", " raise SyntaxError(\"Please check your setting for typos and/or capitalization\")" ] }, { "cell_type": "markdown", "metadata": { "id": "vXWLRdgK1ga4" }, "source": [ "# User Settings\n", "In this section, you'll set your OpenAI API Key (for use with the OpenAI model), configure your environment/files for upload, and upload those files." ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "id": "9GfShBqC1ga5" }, "outputs": [], "source": [ "# Run this cell and enter your OpenAI API key when prompted\n", "openai_api_key = getpass()" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "id": "Y13LXQa81ga6" }, "outputs": [], "source": [ "# Model name\n", "mdl_name = 'gpt-3.5-turbo-0301'" ] }, { "cell_type": "markdown", "metadata": { "id": "UGHgB0lu1ga7" }, "source": [ "## Define Your Document Source\n", "You may upload your files directly from your computer, or you may choose to do so via your Google Drive. Below, you will find instructions for both methods.\n", "\n", "For either model, begin by setting the `upload_setting` variable to:\n", "* `'Local Drive'` - if you have files that are on your own computer (locally), or\n", "* `'Google Drive'` - if you have files that are stored on Google Drive\n", "\n", "e.g.,\n", "`upload_setting='Google Drive'`.\n", "Don't forget the quotes around your selection!" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "id": "RC8Rpo3G1ga7" }, "outputs": [], "source": [ "## Settings for upload: via local drive or Google Drive\n", "### Please input either \"Google Drive\" or \"Local Drive\" into the empty string\n", "\n", "upload_setting = 'Local Drive'\n", "#upload_setting = 'Google Drive'\n", "#upload_setting = 'Local Drive'" ] }, { "cell_type": "markdown", "metadata": { "id": "A5PTo3lB1ga8" }, "source": [ "

Before Continuing - Make sure you have input your choice of upload into the `upload_setting`` variable above (Options: \"Local Drive\" or \"Google Drive\") as described in the above instructions.

" ] }, { "cell_type": "markdown", "metadata": { "id": "JxrCzAIB1ga8" }, "source": [ "## Upload your Files\n", "Now, you'll upload your files. When you run the below code cell, you'll be able to follow the instructions for local or Google Drive upload described here. If you would like to use our example document (Robert Frost's \"The Road Not Taken\", you can download the file from [this link](https://drive.google.com/drive/folders/1wpEoGACUqyNRYa4zBZeNkqcLJrGQbA53?usp=sharing) and upload via the instructions above.\n", "\n", "**If you selected **\"Local Drive\"** :**\n", "> If you selected Local Drive, you'll need to start by selecting your local files. Run the code cell below. Once the icon appears, click the \"Choose File\". This will direct you to your computer's local drive. Select the file you would like to upload as context. The files will appear in the right sidebar. Then follow the rest of the steps in the \"Uploading Your files (Local Drive and Google Drive)\" below.\n", "\n", "**If you selected **\"Google Drive\"**: **\n", "> If you selected Google Drive, you'll need to start by allowing access to your Google Drive. Run the code cell below. You will be redirected to a window where you will allow access to your Google Drive by logging into your Google Account. Your Drive will appear as a folder in the left side panel. Navigate through your Google Drive until you've found the file that you'd like to upload.\n", "\n", "Your files are now accessible to the code." ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "id": "mI7wh4ts1ga9" }, "outputs": [ { "ename": "ModuleNotFoundError", "evalue": "No module named 'google'", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mModuleNotFoundError\u001b[0m Traceback (most recent call last)", "Cell \u001b[0;32mIn[10], line 2\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[38;5;66;03m# Run this cell then following the instructions to upload your file\u001b[39;00m\n\u001b[0;32m----> 2\u001b[0m \u001b[43msetup_drives\u001b[49m\u001b[43m(\u001b[49m\u001b[43mupload_setting\u001b[49m\u001b[43m)\u001b[49m\n", "Cell \u001b[0;32mIn[2], line 7\u001b[0m, in \u001b[0;36msetup_drives\u001b[0;34m(upload_set)\u001b[0m\n\u001b[1;32m 5\u001b[0m \u001b[38;5;66;03m# Colab file upload module\u001b[39;00m\n\u001b[1;32m 6\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m upload_set \u001b[38;5;241m==\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mlocal drive\u001b[39m\u001b[38;5;124m\"\u001b[39m:\n\u001b[0;32m----> 7\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mgoogle\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mcolab\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m files\n\u001b[1;32m 8\u001b[0m uploaded \u001b[38;5;241m=\u001b[39m files\u001b[38;5;241m.\u001b[39mupload()\n\u001b[1;32m 9\u001b[0m \u001b[38;5;28;01melif\u001b[39;00m upload_set \u001b[38;5;241m==\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mgoogle drive\u001b[39m\u001b[38;5;124m\"\u001b[39m:\n\u001b[1;32m 10\u001b[0m \u001b[38;5;66;03m# Mount a Google Drive\u001b[39;00m\n", "\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'google'" ] } ], "source": [ "# Run this cell then following the instructions to upload your file\n", "setup_drives(upload_setting)" ] }, { "cell_type": "markdown", "metadata": { "id": "6JF_7_uz1ga9" }, "source": [ "## Setup file path\n", "Now that you've make your files accessible, we need to select the files of interest. To do this, you'll use the Files pane on the left, following the instructions below to get the filepath. Then, you'll paste the filepath between the quotes below to define the `file_path` variable." ] }, { "cell_type": "markdown", "metadata": { "id": "ffV4STf31ga9" }, "source": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Step 1. Navigate to the three dots to the right
of your file name and click them. 		Step 2. Once the dropdown appears,
select \"Copy Path.\" 		Step 3. Paste the filepath between the single quotes
in the cell below to
define the file path.
Add your filepath to the cell which follows.
Example:
file_path = 'content/roadnottaken.txt'
\n", "
" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "id": "eHNCWHsa1gbB" }, "outputs": [], "source": [ "## Paste your file path into the empty string('') below\n", "#file_path = '/content/roadnottaken.txt' (example)\n", "\n", "file_path = ''" ] }, { "cell_type": "markdown", "metadata": { "id": "ZpVkhIzF1gbB" }, "source": [ "Congratulations! You've finished with the setup! From here, you can now run the rest of the cells to set up your vector store and begin prompting!" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true, "id": "_Oq5MVWD1gbC" }, "source": [ "# Code Setup\n", "Run the following cells to setup the rest of the environment for prompting. In the following section, we set up the computational environment with imported code, setup your API key access to OpenAI, and loading access to your language model. Note that the following cells may take a long time to run." ] }, { "cell_type": "markdown", "metadata": { "id": "34fA_2Xd1gbC" }, "source": [ "## Library installation and loading\n", "The following `pip install` code should be run if you're using Google Colab, or otherwise do not have a computational environment (e.g., _venv_, _conda virtual environment_, _Docker, Singularity, or other container_) with these packages installed." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "Mtl_n3dW1gbC" }, "outputs": [], "source": [ "# run this code if you're using Google Colab or don't have these packages installed in your computing environment\n", "! pip install -q langchain openai gradio numpy chromadb tiktoken unstructured pdf2image" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "id": "4mgqmYzl1gbC" }, "outputs": [], "source": [ "# import required libraries\n", "import numpy as np\n", "from langchain.text_splitter import CharacterTextSplitter\n", "from langchain.embeddings import OpenAIEmbeddings\n", "import os\n", "from langchain.vectorstores import Chroma\n", "from langchain.document_loaders.unstructured import UnstructuredFileLoader\n", "from langchain.document_loaders import UnstructuredFileLoader\n", "from langchain.chains import RetrievalQA\n", "from langchain.chat_models import ChatOpenAI" ] }, { "cell_type": "markdown", "metadata": { "id": "o2w489BP1gbD" }, "source": [ "## API and model setup\n", "\n", "Use these cells to load the API keys required for this notebook and create a basic OpenAI LLM model. The code below uses the variable you created above when you input your API Key." ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "id": "iFz72KVl1gbD" }, "outputs": [], "source": [ "# Set up OpenAI API Key\n", "os.environ[\"OPENAI_API_KEY\"] = openai_api_key" ] }, { "cell_type": "markdown", "metadata": { "id": "Uw_JJ_hk1gbD" }, "source": [ "# Create a vector store with your document\n", "\n", "With the file path, you can now create a vector store using the document that you uploaded. We expose this creation in case you want to modify the kind of vector store that you're creating. Run the cell below to create the default provided vector store." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "IVGFAOmR1gbD" }, "outputs": [], "source": [ "# Create vector store\n", "\n", "loader = UnstructuredFileLoader(file_path)\n", "documents = loader.load()\n", "\n", "text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)\n", "texts = text_splitter.split_documents(documents)\n", "\n", "embeddings = OpenAIEmbeddings()\n", "\n", "db = Chroma.from_documents(texts, embeddings)\n", "\n", "qa = RetrievalQA.from_chain_type(llm=ChatOpenAI(model_name = mdl_name),\n", " chain_type=\"stuff\", retriever=db.as_retriever())" ] }, { "cell_type": "markdown", "metadata": { "id": "vKrmDKIa1gbD" }, "source": [ "# A guide to prompting for self-study\n", "In this section, we provide a number of different approaches for using AI to help you assess and explain the knowledge of your document. Start by interacting with the model and then try out the rest of the prompts!" ] }, { "cell_type": "markdown", "metadata": { "id": "ny-Z9W5-1gbE" }, "source": [ "## Interact with the model\n", "\n", "Now that your vector store is created, you can begin interacting with the model! Below, we have a comprehensive list of examples using different question types, but feel free to use this code block to experiment with the model.\n", "\n", "Input your prompt into the empty string in the code cell. See example below:\n", "\n", "\n", "\n", "```\n", "query = 'Your Prompt Here'\n", "```\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "ob5l7OoB1gbE" }, "outputs": [], "source": [ "# Experiment with interacting with the model by inputting your own prompts into the empty string below.\n", "\n", "query = ''\n", "query_prefix = \"The uploaded document should serve as the basis for the instructions that follow:\"\n", "answer = qa.run(query_prefix + query)\n", "print(answer)" ] }, { "cell_type": "markdown", "metadata": { "id": "hb4APDNu1gbE" }, "source": [ "## Types of Questions and Prompts\n", "\n", "Below is a comprehensive list of question types and prompt templates designed by our team. There are also example code blocks, where you can see how the model performed with the example and try it for yourself using the prompt template." ] }, { "cell_type": "markdown", "metadata": { "id": "QpGXmb2W1gbE" }, "source": [ "### Multiple Choice\n", "\n", "Prompt: The following text should be used as the basis for the instructions which follow: {context}. Please design a 5 question quiz about {name or reference to context} which reflects the learning objectives: {list of learning objectives}. The questions should be multiple choice. If I get an answer wrong, provide me with an explanation of why it was incorrect, and then give me additional chances to respond until I get the correct choice. Explain why the correct choice is right." ] }, { "cell_type": "code", "execution_count": 30, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "6bvn218_1gbE", "outputId": "5180f5da-7557-4e3f-8963-55209833d37f" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "WARNING:chromadb.db.index.hnswlib:Number of requested results 4 is greater than number of elements in index 1, updating n_results = 1\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Great, here's your first question:\n", "\n", "1. Who is the narrator in Robert Frost's \"The Road Not Taken\"?\n", "\n", "A) A young boy\n", "B) An old man\n", "C) A traveler\n", "D) None of the above\n" ] } ], "source": [ "# Multiple choice code example\n", "query = \"\"\"Please design a 5 question quiz about Robert Frost's \"Road Not Taken\" which reflects the learning objectives:\n", "1. Identify the key elements of the poem: narrator, setting, and underlying message.\n", "2. Understand the literary devices used in poetry and their purposes. The questions should be multiple choice.\n", "Provide one question at a time, and wait for my response before providing me with feedback.\n", "If I get an answer wrong, provide me with an explanation of why it was incorrect, and then give me additional\n", "chances to respond until I get the correct choice. Explain why the correct choice is right. \"\"\"\n", "\n", "query_prefix = \"The uploaded document should serve as the basis for the instructions that follow:\"\n", "answer = qa.run(query_prefix + query)\n", "print(answer)" ] }, { "cell_type": "markdown", "metadata": { "id": "NDKq6cfy1gbF" }, "source": [ "### Short Answer\n", "\n", "Prompt: Please design a 5-question quiz about {context} which reflects the learning objectives: {list of learning objectives}. The questions should be short answer. Expect the correct answers to be {anticipated length} long. If I get any part of the answer wrong, provide me with an explanation of why it was incorrect, and then give me additional chances to respond until I get the correct choice." ] }, { "cell_type": "code", "execution_count": 31, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "-LwOHRym1gbG", "outputId": "24e5ef21-1395-4bc8-c082-fef628d10d12" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "WARNING:chromadb.db.index.hnswlib:Number of requested results 4 is greater than number of elements in index 1, updating n_results = 1\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Great! Here is the first question:\n", "\n", "1. Who is the narrator of \"The Road Not Taken\"?\n" ] } ], "source": [ "# Short answer code example\n", "query = \"\"\" Please design a 5-question quiz about Robert Frost's\n", "\"Road Not Taken\" which reflects the learning objectives:\n", "1. Identify the key elements of the poem: narrator, setting, and underlying message.\n", "2. Understand the literary devices used in poetry and their purposes.\n", "The questions should be short answer. Expect the correct answers to be\n", "1-2 sentences long. Provide one question at a time, and wait for my response before providing me with feedback.\n", "If I get any part of the answer wrong,\n", "provide me with an explanation of why it was incorrect,\n", "and then give me additional chances to respond until I get the correct choice. \"\"\"\n", "\n", "answer = qa.run(query_prefix + query)\n", "print(answer)" ] }, { "cell_type": "markdown", "metadata": { "id": "p2f7ePoa1gbH" }, "source": [ "### Fill-in-the-blank\n", "\n", "Prompt: Create a 5 question fill in the blank quiz refrencing {context}. The quiz should reflect the learning objectives: {learning objectives}. Please prompt me one question at a time and proceed when I answer correctly. If I answer incorrectly, please explain why my answer is incorrect." ] }, { "cell_type": "code", "execution_count": 32, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "MU829Vua1gbH", "outputId": "97b540ad-29c4-433d-ff17-25ae686a91cb" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "WARNING:chromadb.db.index.hnswlib:Number of requested results 4 is greater than number of elements in index 1, updating n_results = 1\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Great! Here's your first question:\n", "\n", "1. Who is the author of \"The Road Not Taken\"?\n", "\n", "Please fill in the blank with the correct answer.\n" ] } ], "source": [ "# Fill in the blank code example\n", "query = \"\"\" Create a 5 question fill in the blank quiz refrencing Robert Frost's \"The Road Not Taken.\"\n", "The quiz should reflect the learning objectives:\n", "1. Identify the key elements of the poem: narrator, setting, and underlying message.\n", "2. Understand the literary devices used in poetry and their purposes.\n", "Please prompt me one question at a time and proceed when I answer correctly.\n", "If I answer incorrectly, please explain why my answer is incorrect. \"\"\"\n", "\n", "answer = qa.run(query_prefix + query)\n", "print(answer)" ] }, { "cell_type": "markdown", "metadata": { "id": "Nmb9cMJV1gbH" }, "source": [ "### Sequencing\n", "\n", "Prompt: Please develop a 5 question questionnaire that will ask me to recall the steps involved in the following learning objectives in regard to {context}: {learning objectives}. After I respond, explain their sequence to me." ] }, { "cell_type": "code", "execution_count": 33, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "xhNfe1-y1gbI", "outputId": "f126a1bc-12d1-400a-bb13-5cef38c3dcdf" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "WARNING:chromadb.db.index.hnswlib:Number of requested results 4 is greater than number of elements in index 1, updating n_results = 1\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Great! Here's your first question:\n", "\n", "1. What are the three key elements of Robert Frost's poem \"The Road Not Taken\"?\n", "\n", "Please answer and I will provide the next question.\n" ] } ], "source": [ "# Sequence example\n", "query = \"\"\" Please develop a 5 question questionnaire that will ask me to recall the steps involved in the following learning objectives in regard to Robert Frost's \"The Road Not Taken\":\n", "1. Identify the key elements of the poem: narrator, setting, and underlying message.\n", "2. Understand the literary devices used in poetry and their purposes.\n", "Provide one question at a time, and wait for my response before providing me with feedback.\n", "After I respond, explain their sequence to me.\"\"\"\n", "\n", "answer = qa.run(query_prefix + query)\n", "print(answer)" ] }, { "cell_type": "markdown", "metadata": { "id": "5hYJgb8m1gbI" }, "source": [ "### Relationships/drawing connections\n", "\n", "Prompt: Please design a 5 question quiz that asks me to explain the relationships that exist within the following learning objectives, referencing {context}: {learning objectives}." ] }, { "cell_type": "code", "execution_count": 34, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "xaqrHIYH1gbI", "outputId": "43421d21-c14a-4bfe-be6d-18bc085f29db" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "WARNING:chromadb.db.index.hnswlib:Number of requested results 4 is greater than number of elements in index 1, updating n_results = 1\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Great! Here's your first question: \n", "\n", "1. What are the three key elements of \"The Road Not Taken\" that you should identify?\n" ] } ], "source": [ "# Relationships example\n", "query = \"\"\" Please design a 5 question quiz that asks me to explain the relationships that exist within the following learning objectives, referencing Robert Frost's \"The Road Not Taken\":\n", "1. Identify the key elements of the poem: narrator, setting, and underlying message.\n", "2. Understand the literary devices used in poetry and their purposes.\n", "Provide one question at a time, and wait for my response before providing me with feedback.\"\"\"\n", "\n", "answer = qa.run(query_prefix + query)\n", "print(answer)" ] }, { "cell_type": "markdown", "metadata": { "id": "U4MQSksC1gbO" }, "source": [ "### Concepts and Definitions\n", "\n", "Prompt: Design a 5 question quiz that asks me about definitions related to the following learning objectives: {learning objectives} - based on {context}\".\n", "Once I write out my response, provide me with your own response, highlighting why my answer is correct or incorrect." ] }, { "cell_type": "code", "execution_count": 35, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "cPUWAznY1gbP", "outputId": "44f5d3e4-e2ce-4eff-be02-9f3012ef3adf" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "WARNING:chromadb.db.index.hnswlib:Number of requested results 4 is greater than number of elements in index 1, updating n_results = 1\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Great, here's your first question: \n", "\n", "1. Who is the narrator of Robert Frost's poem \"The Road Not Taken\"? \n", "\n", "Take your time and let me know your answer.\n" ] } ], "source": [ "# Concepts and definitions example\n", "query = \"\"\" Design a 5 question quiz that asks me about definitions related to the following learning objectives:\n", "1. Identify the key elements of the poem: narrator, setting, and underlying message, and\n", "2. Understand the literary devices used in poetry and their purposes - based on Robert Frost's \"The Road Not Taken\".\n", "Provide one question at a time, and wait for my response before providing me with feedback.\n", "Once I write out my response, provide me with your own response, highlighting why my answer is correct or incorrect.\"\"\"\n", "\n", "answer = qa.run(query_prefix + query)\n", "print(answer)" ] }, { "cell_type": "markdown", "metadata": { "id": "tvjkAx1v1gbQ" }, "source": [ "### Real Word Examples\n", "\n", "Prompt: Demonstrate how {context} can be applied to solve a real-world problem related to the following learning objectives: {learning objectives}. Ask me questions regarding this theory/concept." ] }, { "cell_type": "code", "execution_count": 36, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "jqRN7h0N1gbQ", "outputId": "998b0845-d973-4e08-def9-a8e566f994b3" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "WARNING:chromadb.db.index.hnswlib:Number of requested results 4 is greater than number of elements in index 1, updating n_results = 1\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "What are the key elements of Robert Frost's poem \"The Road Not Taken\"?\n" ] } ], "source": [ "# Real word example\n", "query = \"\"\" Demonstrate how Robert Frost’s “The Road Not Taken” can be applied to solve a real-world problem related to the following learning objectives:\n", "1. Identify the key elements of the poem: narrator, setting, and underlying message.\n", "2. Understand the literary devices used in poetry and their purposes.\n", "Ask me questions regarding this theory/concept. Provide one question at a time, and wait for my response before providing me with feedback.\"\"\"\n", "\n", "answer = qa.run(query_prefix + query)\n", "print(answer)" ] }, { "cell_type": "markdown", "metadata": { "id": "6yLvlLNK1gbR" }, "source": [ "### Randomized Question Types\n", "\n", "Prompt: Please generate a high-quality assessment consisting of 5 varying questions, each of different types (open-ended, multiple choice, etc.), to determine if I achieved the following learning objectives in regards to {context}: {learning objectives}. If I answer incorrectly for any of the questions, please explain why my answer is incorrect." ] }, { "cell_type": "code", "execution_count": 40, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "Ht823-F-1gbR", "outputId": "771958ae-0e59-47b7-e750-376615575b30" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "WARNING:chromadb.db.index.hnswlib:Number of requested results 4 is greater than number of elements in index 1, updating n_results = 1\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Question 1 (Open-ended): \n", "Name the key elements of Frost's poem, \"The Road not Taken.\"\n" ] } ], "source": [ "# Randomized question types\n", "query = \"\"\" Please generate a high-quality assessment consisting of 5 varying questions,\n", "each of different types (open-ended, multiple choice, etc.),\n", "to determine if I achieved the following learning objectives in regards to Robert Frost’s “The Road not Taken\":\n", "1. Identify the key elements of the poem: narrator, setting, and underlying message.\n", "2. Understand the literary devices used in poetry and their purposes.\n", "When you provide the 5 questions, only send one at a time, wait for my response, then provide me with feedback.\n", "If I answer incorrectly for any of the questions, please explain why my answer is incorrect.\"\"\"\n", "\n", "answer = qa.run(query_prefix + query)\n", "print(answer)" ] }, { "cell_type": "markdown", "metadata": { "id": "fmz3RrEn1gbR" }, "source": [ "### Quantiative evaluation the correctness of a student's answer\n", "\n", "Prompt: (A continuation of the previous chat) Please generate the main points of the student’s answer to the previous question, and evaluate on a scale of 1 to 5 how comprehensive the student’s answer was in relation to the learning objectives, and explain why he or she received this rating, including what was missed in his or her answer if the student’s answer wasn’t complete.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "EkTzueGw1gbS" }, "outputs": [], "source": [ "# qualitative evaluation\n", "qualitative_query = \"\"\" Please generate the main points of the student’s answer to the previous question,\n", " and evaluate on a scale of 1 to 5 how comprehensive the student’s answer was in relation to the learning objectives,\n", " and explain why he or she received this rating, including what was missed in his or her answer if the student’s answer wasn’t complete.\"\"\"\n", "\n", "# Note that this uses the previous result and query in the context\n", "answer = qa.run(query_prefix + query)\n", "print(answer)" ] } ], "metadata": { "colab": { "include_colab_link": true, "provenance": [] }, "kernelspec": { "display_name": "Python 3.11.5 64-bit", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.5" }, "vscode": { "interpreter": { "hash": "b0fa6594d8f4cbf19f97940f81e996739fb7646882a419484c72d19e05852a7e" } } }, "nbformat": 4, "nbformat_minor": 0 }