{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "90e47f09-34bc-42ef-83bf-3da9e847ec57",
    "_uuid": "64b39254-ee10-411b-87a0-16020f366665",
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "trusted": true
   },
   "source": [
    "*Date:* April 2025\n",
    "\n",
    "*Author:* Marco Mongi\n",
    "\n",
    "This notebook is the Capstone Project for the 5-Day Gen AI Intensive Course, by Google and Kaggle."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Use case: Interactive Adventure Generator\n",
    "The main goal of the project is to **entertain the user**. \n",
    "\n",
    "This is achieved by creating an **interactive adventure generator**, where a Large Language Model (LLM) creates a story and in certain points lets the player choose what to do next. What the player decides affects the progression of the story, as it is dinamically generated by the LLM based on the player's choices, which determines if the ending is good, bad or neutral.\n",
    "\n",
    "**Generative AI is well suited to achieve this goal, as it has unlimited creative potential**, and can create infinite stories to entertain the user. \n",
    "\n",
    "In order to provide a better user experience, a **user interface** made with Gradio allows to interact easily with the LLM and configure some settings. Also, for a more natural interaction, bidirectional **voice interaction** was implemented."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Implementation overview\n",
    "* **Model:** Gemini 2.0 flash is used because of it's fast-response capabilities and long context window\n",
    "* **Prompt:** a system prompt is defined to control how the LLM behaves. The user can modify this prompt by specifying the language (there is a variable containing the language which is injected to the prompt) and custom instructions that modify the story's setting according to user's preferences.\n",
    "* **Agentic Flow:** LangChain was used to define a flow where the narrator (LLM) and the player (user) nodes interact. Conditional edges are defined to end the story, either from the user node (when the user types \"exit\") or the narrator node (the story ends, and the string \"The End.\" is detected).\n",
    "* **Text to Speech:** Piper is used to convert the narrator's text to speech, as it is lightweight and natural sounding. Source: https://github.com/rhasspy/piper\n",
    "* **Speech to Text:** The chosen Gemini model is multimodal, so it can accept audio input, but for this application it is preferred to process audio locally and send the transcribed text to Gemini. OpenAI's Whisper model is used to convert the user's voice to text. The small model's variant is used, as it is faster to run and provides good results. Source: https://github.com/openai/whisper\n",
    "* **Languages:** english and spanish languages are supported, in text interaction, speech recognition and spoken language\n",
    "* **Temperature control:** the user can modify the LLM's temperature setting with a slider in the user interface\n",
    "* **User Interface:** The user can interact with the LLM using a familiar chatbot interface, using text or voice. Similarly, the narrator's output can be read or listened (automatic playback can be enabled). The user can also select the language, modify the LLM's temperature, add custom instructions or start a new story, everything from the User Interface."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "c58271e7-2a8a-46ff-82fa-e2d5bb8f7784",
    "_uuid": "a31dfd36-aaba-474a-b899-1409f8ccea7f",
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "trusted": true
   },
   "source": [
    "# List of gen AI Capabilities used in this project\n",
    "\n",
    "The following table lists all the Generative AI capabilities that this project makes use of, with a brief description of the way it is implemented in the project"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Capability          | How is implemented                                                                                                                                                                                     |\n",
    "|---------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n",
    "| Few-shot prompting  | The prompt includes several example scenarios (Example 1–3) that guide the model’s behavior when generating the narrative.                                                                             |\n",
    "| Audio understanding | User speech is captured via the multimodal textbox and then transcribed into text using Whisper, enabling seamless speech-to-text interaction.                                                         |\n",
    "| Long context window | The system passes the entire conversation history to the LLM on each turn, allowing the model to maintain context throughout a multi-turn interaction.                                                 |\n",
    "| Agents              | The conversation flow is structured using a simple agent model (with distinct “player” and “narrator” nodes and conditional transitions) that directs how the dialogue progresses based on user input. |\n",
    "| Context caching     | Conversation history is stored in a session state variable, effectively caching context within each independent story session for coherent multi-turn interactions.                                    |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">Note: Some structured output/controlled generation is performed via prompt engineering, such as controlling the temperature value, the language used by the LLM and custom user instructions (by injecting variables in the prompt), the tone and narrative of the story, and ending it with the precise words \"The End.\", but as it a very simple output control, it is not listed in the table above"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Implementation (code)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-04-13T18:12:56.429396Z",
     "iopub.status.busy": "2025-04-13T18:12:56.429075Z",
     "iopub.status.idle": "2025-04-13T18:12:56.438943Z",
     "shell.execute_reply": "2025-04-13T18:12:56.438048Z",
     "shell.execute_reply.started": "2025-04-13T18:12:56.429372Z"
    },
    "trusted": true
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install dependencies\n",
    "The required packages that are not installed by default in the environment need to be installed manually\n",
    "\n",
    ">Note: langchain-google-genai has a dependency problem with async-timeout and shows an error, but this does not affect the code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-04-13T18:12:56.440992Z",
     "iopub.status.busy": "2025-04-13T18:12:56.440617Z",
     "iopub.status.idle": "2025-04-13T18:15:02.520123Z",
     "shell.execute_reply": "2025-04-13T18:15:02.518749Z",
     "shell.execute_reply.started": "2025-04-13T18:12:56.440952Z"
    },
    "trusted": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m43.5/43.5 kB\u001b[0m \u001b[31m1.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m138.0/138.0 kB\u001b[0m \u001b[31m7.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m42.0/42.0 kB\u001b[0m \u001b[31m2.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.4/1.4 MB\u001b[0m \u001b[31m43.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m423.3/423.3 kB\u001b[0m \u001b[31m20.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m42.0/42.0 kB\u001b[0m \u001b[31m2.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m47.2/47.2 kB\u001b[0m \u001b[31m3.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m223.6/223.6 kB\u001b[0m \u001b[31m13.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
      "google-generativeai 0.8.4 requires google-ai-generativelanguage==0.6.15, but you have google-ai-generativelanguage 0.6.17 which is incompatible.\u001b[0m\u001b[31m\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m800.5/800.5 kB\u001b[0m \u001b[31m21.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25h  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
      "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
      "  Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m363.4/363.4 MB\u001b[0m \u001b[31m4.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m0:00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m664.8/664.8 MB\u001b[0m \u001b[31m2.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m0:00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m211.5/211.5 MB\u001b[0m \u001b[31m5.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m0:00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m56.3/56.3 MB\u001b[0m \u001b[31m27.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m127.9/127.9 MB\u001b[0m \u001b[31m10.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m207.5/207.5 MB\u001b[0m \u001b[31m1.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m21.1/21.1 MB\u001b[0m \u001b[31m30.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25h  Building wheel for openai-whisper (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
      "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
      "pylibcugraph-cu12 24.12.0 requires pylibraft-cu12==24.12.*, but you have pylibraft-cu12 25.2.0 which is incompatible.\n",
      "pylibcugraph-cu12 24.12.0 requires rmm-cu12==24.12.*, but you have rmm-cu12 25.2.0 which is incompatible.\u001b[0m\u001b[31m\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m46.9/46.9 MB\u001b[0m \u001b[31m34.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m322.2/322.2 kB\u001b[0m \u001b[31m19.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m95.2/95.2 kB\u001b[0m \u001b[31m6.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m16.0/16.0 MB\u001b[0m \u001b[31m79.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m25.0/25.0 MB\u001b[0m \u001b[31m66.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m11.4/11.4 MB\u001b[0m \u001b[31m70.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m:01\u001b[0mm\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m72.0/72.0 kB\u001b[0m \u001b[31m4.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m62.4/62.4 kB\u001b[0m \u001b[31m3.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m46.0/46.0 kB\u001b[0m \u001b[31m2.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m86.8/86.8 kB\u001b[0m \u001b[31m5.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h"
     ]
    }
   ],
   "source": [
    "!pip install -qU 'langgraph==0.3.21' 'langchain-google-genai==2.1.2' --retries 3\n",
    "!pip install -q openai-whisper\n",
    "!pip install -q --upgrade gradio piper-tts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import Google API Key\n",
    "As the models runs on Google remote servers, in order to access them you must authenticate by providing an API Key.\n",
    "\n",
    "Before running the following cell, your API key must be stored it in a [Kaggle secret](https://www.kaggle.com/discussions/product-feedback/114053) named `GOOGLE_API_KEY`.\n",
    "\n",
    "This [documentation](https://ai.google.dev/gemini-api/docs/api-key) provides detailed instructions on how to generate your own API Key (if you don't already have one).\n",
    "\n",
    "Once you have your Key, go to \"Add-ons\" on the top of the Kaggle Notebook interface and go to \"Secrets\". There you press the \"Add Secret\" button. In LABEL, write GOOGLE_API_KEY (this name should not be changed), and in VALUE paste your API Key\n",
    "\n",
    ">Note: API Keys are for personal use, and should not be shared"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-04-13T18:15:02.522298Z",
     "iopub.status.busy": "2025-04-13T18:15:02.521878Z",
     "iopub.status.idle": "2025-04-13T18:15:02.731692Z",
     "shell.execute_reply": "2025-04-13T18:15:02.730779Z",
     "shell.execute_reply.started": "2025-04-13T18:15:02.522231Z"
    },
    "trusted": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "from kaggle_secrets import UserSecretsClient\n",
    "\n",
    "GOOGLE_API_KEY = UserSecretsClient().get_secret(\"GOOGLE_API_KEY\")\n",
    "os.environ[\"GOOGLE_API_KEY\"] = GOOGLE_API_KEY"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Chatbot configuration\n",
    "Defines initial prompts, examples, and generates the first message using the LLM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-04-13T18:15:02.734450Z",
     "iopub.status.busy": "2025-04-13T18:15:02.734111Z",
     "iopub.status.idle": "2025-04-13T18:15:05.600984Z",
     "shell.execute_reply": "2025-04-13T18:15:05.600108Z",
     "shell.execute_reply.started": "2025-04-13T18:15:02.734424Z"
    },
    "trusted": true
   },
   "outputs": [],
   "source": [
    "from langchain_google_genai import ChatGoogleGenerativeAI\n",
    "from langchain.schema import SystemMessage, HumanMessage, AIMessage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-04-13T18:15:05.602489Z",
     "iopub.status.busy": "2025-04-13T18:15:05.601957Z",
     "iopub.status.idle": "2025-04-13T18:15:07.310465Z",
     "shell.execute_reply": "2025-04-13T18:15:07.309749Z",
     "shell.execute_reply.started": "2025-04-13T18:15:05.602464Z"
    },
    "trusted": true
   },
   "outputs": [],
   "source": [
    "# Initial settings\n",
    "TEMPERATURE = 1                # Float between 0-2\n",
    "LANGUAGE = \"English\"           # Available \"English\" and \"Español\"\n",
    "\n",
    "# System Prompt\n",
    "NARRATOR_SYS_MSG = (\n",
    "    \"system\",\n",
    "    \"You are the narrator of an interactive story where the player's choices directly influence the progression and outcome of the narrative. \"\n",
    "    \"Begin with an engaging introduction: set the stage with vivid, sensory details; describe the setting, introduce key characters, and hint at the main conflict. \"\n",
    "    \"Speak directly to the player using 'you' to draw them into the story. As the story unfolds, organically introduce decision points where you ask the player what will he do. \"\n",
    "    \"Reflect the consequences of the player's choices, leading to multiple possible endings. \"\n",
    "    \"If the narrative reaches a point where the character dies, end with 'The End.' If the story concludes naturally, finish with 'The End.'\"\n",
    ")\n",
    "\n",
    "# First message prompt: story generation guidelines\n",
    "INTRO_STORY_PROMPT = (\n",
    "    \"Generate a welcome message for the interactive story with a new setting, introducing the player's role, environment, \"\n",
    "    \"key characters, and hinting at the main conflict. End with a question asking what the player will do. Do not present possible alternatives, let the player create his own.\"\n",
    ")\n",
    "LANGUAGE_PROMPT = f\"Answer always using the language {LANGUAGE}\"\n",
    "EXAMPLE1 = (\"Example 1: You are a warrior in a medieval town, your sister recently died at the hands of an evil sorcerer. \"\n",
    "            \"You are currently heading to the market to complete an errand for a friend, where you find a stranger sitting on a table and mysteriously looking at you. What will you do?\")\n",
    "EXAMPLE2 = (\"Example 2: You are a pirate in Blackbeard's ship. The morning was going as usual, with the salty odor and calm waters, \"\n",
    "            \"until you hear a stomp and see a giant tentacle going into the water. You ask another tripulant but he didn't hear or see anything. What will you do?\")\n",
    "EXAMPLE3 = (\"Example 3: You are a spaceship captain in the hunt for the infamous thief Lauren DeHugh, your crew follows you with pride and loyalty, \"\n",
    "            \"but recently the moods have been weird, you suspect that the new passenger may have something to do, but it could also be nothing. \"\n",
    "            \"Currently, you need to check the map and then you have a few minutes of spare time. What do you want to do?\")\n",
    "\n",
    "DYNAMIC_PROMPT = INTRO_STORY_PROMPT + EXAMPLE1 + EXAMPLE2 + EXAMPLE3 + LANGUAGE_PROMPT\n",
    "\n",
    "\n",
    "# Instantiate the initial LLM and generate the first message\n",
    "llm = ChatGoogleGenerativeAI(model=\"gemini-2.0-flash\", temperature=TEMPERATURE)\n",
    "first_message = llm.invoke([\n",
    "    SystemMessage(content=NARRATOR_SYS_MSG[1]),\n",
    "    HumanMessage(content=DYNAMIC_PROMPT)\n",
    "]).content\n",
    "initial_chat_history = [(\"\", first_message)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Piper TTS Configuration\n",
    "Configures the Piper voices, forcing redownload if needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-04-13T18:15:07.311556Z",
     "iopub.status.busy": "2025-04-13T18:15:07.311317Z",
     "iopub.status.idle": "2025-04-13T18:15:07.359114Z",
     "shell.execute_reply": "2025-04-13T18:15:07.358430Z",
     "shell.execute_reply.started": "2025-04-13T18:15:07.311537Z"
    },
    "trusted": true
   },
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "import tempfile\n",
    "from piper.voice import PiperVoice\n",
    "from piper.download import get_voices, ensure_voice_exists, find_voice\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-04-13T18:15:07.360211Z",
     "iopub.status.busy": "2025-04-13T18:15:07.360002Z",
     "iopub.status.idle": "2025-04-13T18:15:11.244156Z",
     "shell.execute_reply": "2025-04-13T18:15:11.243426Z",
     "shell.execute_reply.started": "2025-04-13T18:15:07.360195Z"
    },
    "trusted": true
   },
   "outputs": [],
   "source": [
    "download_dir = Path.home() / \".local/share/piper-tts/piper-voices\"\n",
    "\n",
    "voices_info = get_voices(download_dir)\n",
    "piper_voice_names = {\n",
    "    \"Español\": \"es_ES-carlfm-x_low\",\n",
    "    \"English\": \"en_US-joe-medium\"\n",
    "}\n",
    "# Force redownload of corrupted configurations\n",
    "for voice_name in piper_voice_names.values():\n",
    "    config_file = download_dir / f\"{voice_name}.onnx.json\"\n",
    "    if config_file.exists():\n",
    "        config_file.unlink()\n",
    "voices = {}\n",
    "for lang, voice_name in piper_voice_names.items():\n",
    "    ensure_voice_exists(voice_name, [download_dir], download_dir, voices_info)\n",
    "    model_path, config_path = find_voice(voice_name, [download_dir])\n",
    "    voices[lang] = PiperVoice.load(model_path, config_path=config_path)\n",
    "\n",
    "def text_to_speech(lang: str, text: str) -> str:\n",
    "    \"\"\"Convert text to speech using Piper TTS and return a file path.\"\"\"\n",
    "    voice = voices[lang]\n",
    "    raw_chunks = voice.synthesize_stream_raw(text)\n",
    "    audio_bytes = b\"\".join(raw_chunks)\n",
    "    audio_array = np.frombuffer(audio_bytes, dtype=np.int16)\n",
    "    temp_audio = tempfile.NamedTemporaryFile(suffix=\".wav\", delete=False)\n",
    "    sf.write(temp_audio.name, audio_array, voice.config.sample_rate, subtype=\"PCM_16\")\n",
    "    return temp_audio.name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Whisper speech-to-text Configuration\n",
    "Sets up the Whisper model for transcribing audio input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-04-13T18:15:11.245361Z",
     "iopub.status.busy": "2025-04-13T18:15:11.245059Z",
     "iopub.status.idle": "2025-04-13T18:15:16.214965Z",
     "shell.execute_reply": "2025-04-13T18:15:16.214097Z",
     "shell.execute_reply.started": "2025-04-13T18:15:11.245332Z"
    },
    "trusted": true
   },
   "outputs": [],
   "source": [
    "import whisper"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-04-13T18:15:16.216413Z",
     "iopub.status.busy": "2025-04-13T18:15:16.215961Z",
     "iopub.status.idle": "2025-04-13T18:15:35.181629Z",
     "shell.execute_reply": "2025-04-13T18:15:35.180706Z",
     "shell.execute_reply.started": "2025-04-13T18:15:16.216392Z"
    },
    "trusted": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████| 461M/461M [00:13<00:00, 36.4MiB/s]\n"
     ]
    }
   ],
   "source": [
    "model_whisper = whisper.load_model(\"small\")\n",
    "language_codes = {\n",
    "    \"Español\": \"es\",\n",
    "    \"English\": \"en\"\n",
    "}\n",
    "def transcribe_audio(lang: str, audio_path: str) -> str:\n",
    "    \"\"\"Transcribe audio input using Whisper.\"\"\"\n",
    "    code = language_codes[lang]\n",
    "    result = model_whisper.transcribe(audio_path, language=code)\n",
    "    return result[\"text\"].strip()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Integrated Chatbot Generation (Streaming)\n",
    "Processes multimodal input, streams the LLM response, and generates narration audio."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-04-13T18:15:35.184050Z",
     "iopub.status.busy": "2025-04-13T18:15:35.183787Z",
     "iopub.status.idle": "2025-04-13T18:15:39.612950Z",
     "shell.execute_reply": "2025-04-13T18:15:39.612003Z",
     "shell.execute_reply.started": "2025-04-13T18:15:35.184031Z"
    },
    "trusted": true
   },
   "outputs": [],
   "source": [
    "import gradio as gr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-04-13T18:15:39.614763Z",
     "iopub.status.busy": "2025-04-13T18:15:39.614002Z",
     "iopub.status.idle": "2025-04-13T18:15:39.625230Z",
     "shell.execute_reply": "2025-04-13T18:15:39.624290Z",
     "shell.execute_reply.started": "2025-04-13T18:15:39.614734Z"
    },
    "trusted": true
   },
   "outputs": [],
   "source": [
    "def integrated_generate(user_input, selected_language, autoplay_enabled, temperature_value, chat_history):\n",
    "    \"\"\"\n",
    "    Process multimodal user input (text or microphone recording) and stream the LLM-generated narrative.\n",
    "    Returns updated chat history and updated audio component.\n",
    "    \"\"\"\n",
    "    input_text = \"\"\n",
    "    recorded_audio = None\n",
    "    if isinstance(user_input, dict):\n",
    "        input_text = user_input.get(\"text\", \"\").strip()\n",
    "        files = user_input.get(\"files\", [])\n",
    "        if files:\n",
    "            file_item = files[0]\n",
    "            recorded_audio = file_item.get(\"path\", None) if isinstance(file_item, dict) else file_item\n",
    "    else:\n",
    "        input_text = str(user_input).strip()\n",
    "    \n",
    "    final_input = input_text if input_text != \"\" else (transcribe_audio(selected_language, recorded_audio) if recorded_audio else \"\")\n",
    "    \n",
    "    if final_input == \"\":\n",
    "        yield chat_history, chat_history, gr.update(interactive=True, value=\"\"), None\n",
    "        return\n",
    "    if final_input.lower() in {\"q\", \"quit\", \"exit\", \"goodbye\"}:\n",
    "        chat_history = chat_history + [(final_input, \"The End.\")]\n",
    "        yield chat_history, chat_history, gr.update(interactive=False, value=\"\"), None\n",
    "        return\n",
    "\n",
    "    # Build messages for LLM\n",
    "    messages = [SystemMessage(content=NARRATOR_SYS_MSG[1])]\n",
    "    if chat_history and chat_history[0][0] == \"\":\n",
    "        messages.append(AIMessage(content=chat_history[0][1]))\n",
    "    else:\n",
    "        for human_msg, ai_msg in chat_history:\n",
    "            messages.append(HumanMessage(content=human_msg))\n",
    "            messages.append(AIMessage(content=ai_msg))\n",
    "    messages.append(HumanMessage(content=final_input))\n",
    "    chat_history = chat_history + [(final_input, \"\")]\n",
    "    generated_text = \"\"\n",
    "    streaming_llm = ChatGoogleGenerativeAI(model=\"gemini-2.0-flash\", streaming=True, temperature=temperature_value)\n",
    "    for token in streaming_llm.stream(messages):\n",
    "        generated_text += token.content\n",
    "        chat_history[-1] = (final_input, generated_text)\n",
    "        yield chat_history, chat_history, gr.update(interactive=True, value=\"\"), None\n",
    "    end_condition = \"The End.\" in generated_text\n",
    "    narration_audio = text_to_speech(selected_language, generated_text)\n",
    "    yield chat_history, chat_history, gr.update(interactive=not end_condition, value=\"\"), gr.update(autoplay=autoplay_enabled, value=narration_audio)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reset chat function\n",
    "Resets the conversation with updated settings (language, temperature, autoplay, and user preferences)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-04-13T18:15:39.627509Z",
     "iopub.status.busy": "2025-04-13T18:15:39.627120Z",
     "iopub.status.idle": "2025-04-13T18:15:39.649819Z",
     "shell.execute_reply": "2025-04-13T18:15:39.648878Z",
     "shell.execute_reply.started": "2025-04-13T18:15:39.627476Z"
    },
    "trusted": true
   },
   "outputs": [],
   "source": [
    "def reset_chat(selected_language, autoplay_enabled, temperature_value, user_preferences):\n",
    "    \"\"\"\n",
    "    Reset the conversation using new settings and optional user preferences.\n",
    "    Updates the global LANGUAGE and prompt variables accordingly.\n",
    "    \"\"\"\n",
    "    global LANGUAGE, LANGUAGE_PROMPT, DYNAMIC_PROMPT\n",
    "    LANGUAGE = selected_language\n",
    "    LANGUAGE_PROMPT = f\"Answer always using the language {LANGUAGE}\"\n",
    "    prefs_text = f\"The story should take into consideration the following: {user_preferences.strip()}\" if user_preferences.strip() else \"\"\n",
    "    DYNAMIC_PROMPT = INTRO_STORY_PROMPT + EXAMPLE1 + EXAMPLE2 + EXAMPLE3 + LANGUAGE_PROMPT + prefs_text\n",
    "    llm_instance = ChatGoogleGenerativeAI(model=\"gemini-2.0-flash\", temperature=temperature_value)\n",
    "    new_message = llm_instance.invoke([\n",
    "        SystemMessage(content=NARRATOR_SYS_MSG[1]),\n",
    "        HumanMessage(content=DYNAMIC_PROMPT)\n",
    "    ]).content\n",
    "    new_history = [(\"\", new_message)]\n",
    "    initial_audio = text_to_speech(selected_language, new_message)\n",
    "    return new_history, new_history, gr.update(interactive=True, value=\"\"), gr.update(autoplay=autoplay_enabled, value=initial_audio)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gradio User Interface\n",
    "Organized into a full-width chat area on top and a two-row, two-column layout below for input, narration audio, additional settings, and a reset button."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-04-13T18:15:39.650981Z",
     "iopub.status.busy": "2025-04-13T18:15:39.650708Z",
     "iopub.status.idle": "2025-04-13T18:15:39.691149Z",
     "shell.execute_reply": "2025-04-13T18:15:39.690149Z",
     "shell.execute_reply.started": "2025-04-13T18:15:39.650951Z"
    },
    "trusted": true
   },
   "outputs": [],
   "source": [
    "import soundfile as sf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-04-13T18:15:39.692295Z",
     "iopub.status.busy": "2025-04-13T18:15:39.692043Z",
     "iopub.status.idle": "2025-04-13T18:15:44.400896Z",
     "shell.execute_reply": "2025-04-13T18:15:44.400201Z",
     "shell.execute_reply.started": "2025-04-13T18:15:39.692267Z"
    },
    "trusted": true
   },
   "outputs": [],
   "source": [
    "initial_audio = text_to_speech(\"English\", first_message)\n",
    "\n",
    "with gr.Blocks() as demo:\n",
    "    # Upper section: Chat (full width)\n",
    "    gr.Markdown(\"## Interactive Adventure\")\n",
    "    chat_box = gr.Chatbot(value=initial_chat_history, type=\"tuples\")\n",
    "    chat_state = gr.State(initial_chat_history)\n",
    "    \n",
    "    # Lower section: Two rows (first row: 75% user input, 25% narration audio; second row: 75% settings, 25% reset button)\n",
    "    # Row 1: user input and narrator audio\n",
    "    with gr.Row():\n",
    "        with gr.Column(scale=3):\n",
    "            user_input = gr.MultimodalTextbox(\n",
    "                label=\"Your action\",\n",
    "                placeholder=\"Type your action or press the microphone icon\",\n",
    "                sources=[\"microphone\"]\n",
    "            )\n",
    "        with gr.Column(scale=1):\n",
    "            narration_audio = gr.Audio(value=initial_audio, label=\"Narration\", type=\"filepath\", autoplay=False)\n",
    "    \n",
    "    # Row 2: settings and reset button\n",
    "    with gr.Row():\n",
    "        with gr.Column(scale=3):\n",
    "            with gr.Accordion(\"Additional Settings\", open=False):\n",
    "                setting_language = gr.Dropdown(\n",
    "                    choices=[\"Español\", \"English\"],\n",
    "                    value=\"English\",\n",
    "                    label=\"Language\"\n",
    "                )\n",
    "                setting_autoplay = gr.Checkbox(\n",
    "                    value=False,\n",
    "                    label=\"Autoplay Narration\"\n",
    "                )\n",
    "                setting_temperature = gr.Slider(\n",
    "                    minimum=0,\n",
    "                    maximum=2,\n",
    "                    step=0.1,\n",
    "                    value=1,\n",
    "                    label=\"Temperature\"\n",
    "                )\n",
    "                setting_preferences = gr.Textbox(\n",
    "                    label=\"User Preferences\",\n",
    "                    placeholder=\"Write your own guidelines to the story\",\n",
    "                    lines=2\n",
    "                )\n",
    "        with gr.Column(scale=1):\n",
    "            reset_button = gr.Button(\"Start Again\")\n",
    "    \n",
    "    # Trigger submission on Enter in MultimodalTextbox\n",
    "    user_input.submit(\n",
    "        fn=integrated_generate,\n",
    "        inputs=[user_input, setting_language, setting_autoplay, setting_temperature, chat_state],\n",
    "        outputs=[chat_box, chat_state, user_input, narration_audio]\n",
    "    )\n",
    "    \n",
    "    reset_button.click(\n",
    "        fn=reset_chat,\n",
    "        inputs=[setting_language, setting_autoplay, setting_temperature, setting_preferences],\n",
    "        outputs=[chat_box, chat_state, user_input, narration_audio],\n",
    "        queue=False\n",
    "    )\n",
    "    \n",
    "    #demo.launch(share=True, debug=True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**IMPORTANT**\n",
    "\n",
    "To run the app, you must **uncomment the following line**. \n",
    "\n",
    ">Note: It is recommended to access the app by the provided public URL, as microphone input is not allowed inside the Kaggle Notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-04-13T18:15:44.402013Z",
     "iopub.status.busy": "2025-04-13T18:15:44.401751Z",
     "iopub.status.idle": "2025-04-13T18:15:44.405974Z",
     "shell.execute_reply": "2025-04-13T18:15:44.405311Z",
     "shell.execute_reply.started": "2025-04-13T18:15:44.401994Z"
    },
    "trusted": true
   },
   "outputs": [],
   "source": [
    "#demo.launch(share=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Citation\n",
    "Addison Howard, Brenda Flynn, Kinjal Parekh, Myles O'Neill, Nate, and Polong Lin. Gen AI Intensive Course Capstone 2025Q1. https://kaggle.com/competitions/gen-ai-intensive-course-capstone-2025q1, 2025. Kaggle."
   ]
  }
 ],
 "metadata": {
  "kaggle": {
   "accelerator": "none",
   "dataSources": [
    {
     "sourceId": 97258,
     "sourceType": "competition"
    }
   ],
   "dockerImageVersionId": 31012,
   "isGpuEnabled": false,
   "isInternetEnabled": true,
   "language": "python",
   "sourceType": "notebook"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}