notebooks/AIE2_midterm.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "collapsed_sections": [
        "QojysEo6Soqb",
        "8tpilHaaSoSg",
        "p0BhZDfbk2KT"
      ]
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Midterm Certification Challenge: Building and Deploying a RAG Application\n",
        "DUE DATE: Before 4:00 PM PT on May 2 (before next Thursday's class!)\n",
        "\n",
        "You are to record the total time it takes you to complete\n",
        "\n",
        "You have access to all boiler-plate code from the course, and we highly encourage you to leverage it!\n",
        "\n",
        "**Deliverables:**\n",
        "\n",
        "**Build 🏗️**\n",
        "\n",
        "* Data: Meta 10-k Filings\n",
        "* LLM: OpenAI GPT-3.5-turbo\n",
        "* Embedding Model: text-3-embedding small\n",
        "* Infrastructure: LangChain or LlamaIndex (you choose)\n",
        "* Vector Store: Qdrant\n",
        "* Deployment: Chainlit, Hugging Face\n",
        "**Ship 🚢**\n",
        "\n",
        "* Evaluate your answers to the following questions\n",
        "\"What was the total value of 'Cash and cash equivalents' as of December 31, 2023?\"\n",
        "\"Who are Meta's 'Directors' (i.e., members of the Board of Directors)?\"\n",
        "* Record <10 min loom video walkthrough\n",
        "$$ Extra Credit: Baseline retrieval performance w/ RAGAS, change something about your RAG system to improve it, then show the improvement quantitatively!\n",
        "\n",
        "**Share 🚀**\n",
        "* Share lessons not yet learned in #aie2-general"
      ],
      "metadata": {
        "id": "uDsowVwcRyZ8"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Install Dependencies"
      ],
      "metadata": {
        "id": "QojysEo6Soqb"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import nest_asyncio\n",
        "\n",
        "nest_asyncio.apply()"
      ],
      "metadata": {
        "id": "hizlCdZeh1i7"
      },
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "BXSRaRN2RixC"
      },
      "outputs": [],
      "source": [
        "!pip install llama-parse llama_index -qU"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install -qU langchain langchain-core langchain-community langchain-openai unstructured"
      ],
      "metadata": {
        "id": "uZJ6AIUs3c4j"
      },
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install -qU qdrant-client"
      ],
      "metadata": {
        "id": "5quBcn6K39hF"
      },
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Set Environment Variables"
      ],
      "metadata": {
        "id": "8tpilHaaSoSg"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "from getpass import getpass\n",
        "\n",
        "# set openai key\n",
        "os.environ[\"OPENAI_API_KEY\"] = getpass(\"OpenAI API Key:\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AfvigvZfTXjX",
        "outputId": "b6c3f8e5-7701-494f-e6e3-85fed9471d5f"
      },
      "execution_count": 13,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "OpenAI API Key:··········\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# set llama cloud key\n",
        "os.environ[\"LLAMA_CLOUD_API_KEY\"] = getpass(\"Llama Cloud API Key:\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4WVa_areeqb4",
        "outputId": "4433e2ce-9a37-4b5d-975d-9a310710708d"
      },
      "execution_count": 9,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Llama Cloud API Key:··········\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Download the Data"
      ],
      "metadata": {
        "id": "p0BhZDfbk2KT"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# download the data\n",
        "!mkdir 'data'\n",
        "!wget 'https://d18rn0p25nwr6d.cloudfront.net/CIK-0001326801/c7318154-f6ae-4866-89fa-f0c589f2ee3d.pdf' -O 'data/Meta_10k.pdf'"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XJbBJ17zieee",
        "outputId": "1bc31bbd-7139-4f29-ef17-2a591815d76f"
      },
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2024-05-02 18:27:08--  https://d18rn0p25nwr6d.cloudfront.net/CIK-0001326801/c7318154-f6ae-4866-89fa-f0c589f2ee3d.pdf\n",
            "Resolving d18rn0p25nwr6d.cloudfront.net (d18rn0p25nwr6d.cloudfront.net)... 18.154.131.210, 18.154.131.173, 18.154.131.90, ...\n",
            "Connecting to d18rn0p25nwr6d.cloudfront.net (d18rn0p25nwr6d.cloudfront.net)|18.154.131.210|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 2481466 (2.4M) [application/pdf]\n",
            "Saving to: ‘data/Meta_10k.pdf’\n",
            "\n",
            "\rdata/Meta_10k.pdf     0%[                    ]       0  --.-KB/s               \rdata/Meta_10k.pdf   100%[===================>]   2.37M  --.-KB/s    in 0.05s   \n",
            "\n",
            "2024-05-02 18:27:08 (47.1 MB/s) - ‘data/Meta_10k.pdf’ saved [2481466/2481466]\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## RAG with Llama Parse + LangChain RecursiveCharacterTextSplitter"
      ],
      "metadata": {
        "id": "j9I3akvG2AjJ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "First, we'll parse the document using Llama Parse. Then we'll save the llama_parse markdown document so we can use it later."
      ],
      "metadata": {
        "id": "QiAk6mbE6E4L"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# import dependencies\n",
        "from llama_parse import LlamaParse\n",
        "from llama_index.core import SimpleDirectoryReader\n",
        "\n",
        "parsing_instruction = \"\"\"The provided document is an annual report filed by Meta Platforms, Inc. with the Securities and Exchange Commission (SEC).\n",
        "This form provides detailed financial information about the company's performance for a specific year.\n",
        "It includes financial statements, management discussion and analysis, and other relevant disclosures required by the SEC.\n",
        "It contains many tables and some signature pages.\n",
        "\n",
        "Replace the signatures with tables containing the headers for each element.\n",
        "\"\"\"\n",
        "\n",
        "# setup parser\n",
        "parser = LlamaParse(\n",
        "    result_type=\"markdown\",\n",
        "    parsing_instruction=parsing_instruction\n",
        ")\n",
        "\n",
        "# load and parse the documet\n",
        "file_extractor = {\".pdf\": parser}\n",
        "llama_parse_documents = SimpleDirectoryReader(\n",
        "    input_files=['data/Meta_10k.pdf'],\n",
        "    file_extractor=file_extractor\n",
        ").load_data()\n",
        "\n",
        "# save markdown file\n",
        "data_file = \"./data/output.md\"\n",
        "with open(data_file, \"a\") as f:\n",
        "    for doc in llama_parse_documents:\n",
        "        f.write(doc.text + '\\n')"
      ],
      "metadata": {
        "id": "Q9LXiA4U12aM",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "64b30010-24a3-4a5f-a834-e6978857657f"
      },
      "execution_count": 71,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Started parsing the file under job_id f27fcb88-f758-4d41-b1ab-d38b8daf6754\n",
            "...."
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now we'll setup the langchain RAG with Qdrant"
      ],
      "metadata": {
        "id": "JlwmRXZi6PYu"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# import dependencies\n",
        "from langchain.text_splitter import RecursiveCharacterTextSplitter, MarkdownHeaderTextSplitter\n",
        "from langchain_community.vectorstores import Qdrant\n",
        "from langchain_community.document_loaders import DirectoryLoader\n",
        "from langchain_openai.embeddings import OpenAIEmbeddings\n",
        "\n",
        "# load the document\n",
        "loader = DirectoryLoader(path='data/', glob=\"**/*.md\", show_progress=True)\n",
        "documents = loader.load()\n",
        "\n",
        "# split the document into chunks\n",
        "\n",
        "# split markdown headers\n",
        "headers_to_split_on = [\n",
        "    (\"#\", \"Header 1\"),\n",
        "    (\"##\", \"Header 2\"),\n",
        "    (\"###\", \"Header 3\"),\n",
        "]\n",
        "\n",
        "md_text_splitter = MarkdownHeaderTextSplitter(\n",
        "    headers_to_split_on=headers_to_split_on,\n",
        "    strip_headers = False\n",
        ")\n",
        "\n",
        "md_splits = md_text_splitter.split_text(documents[0].page_content)\n",
        "\n",
        "# recursive character text splitter\n",
        "text_splitter = RecursiveCharacterTextSplitter(chunk_size=2500, chunk_overlap=100)\n",
        "docs = text_splitter.split_documents(md_splits)\n",
        "\n",
        "# instantiate embeddings\n",
        "embeddings = OpenAIEmbeddings(model=\"text-embedding-3-small\")\n",
        "\n",
        "# create the vectorstore\n",
        "qdrant_vector_store = Qdrant.from_documents(\n",
        "    documents=docs,\n",
        "    embedding=embeddings,\n",
        "    location=\":memory:\",\n",
        "    collection_name=\"meta_10k\"\n",
        ")\n",
        "\n",
        "# setup our retriever\n",
        "qdrant_retriever = qdrant_vector_store.as_retriever()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Mkq3v2XX5oY0",
        "outputId": "18c9039b-3f22-44b8-a830-2f64c9d8e59b"
      },
      "execution_count": 83,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "100%|██████████| 1/1 [00:16<00:00, 16.71s/it]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Next, we'll setup the RAG Prompt."
      ],
      "metadata": {
        "id": "n5l5oZwq_1zq"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from langchain_core.prompts import ChatPromptTemplate\n",
        "\n",
        "RAG_PROMPT = \"\"\"\n",
        "CONTEXT:\n",
        "{context}\n",
        "\n",
        "QUERY:\n",
        "{question}\n",
        "\n",
        "The provided context is an annual report filed by Meta Platforms, Inc. with the Securities and Exchange Commission (SEC).\n",
        "This form provides detailed financial information about the company's performance for a specific year.\n",
        "It includes financial statements, management discussion and analysis, and other relevant disclosures required by the SEC.\n",
        "It contains many tables and some signature pages. All members of the board need to sign the document.\n",
        "\n",
        "Answer the query above only using the context provided. If you don't know the answer, simply say 'I don't know'.\n",
        "\"\"\"\n",
        "\n",
        "rag_prompt = ChatPromptTemplate.from_template(RAG_PROMPT)"
      ],
      "metadata": {
        "id": "MFvFEItd_4nq"
      },
      "execution_count": 84,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Finally, we create our chain..."
      ],
      "metadata": {
        "id": "SlmhFU4DACib"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from operator import itemgetter\n",
        "from langchain_core.runnables import RunnablePassthrough\n",
        "from langchain_core.output_parsers import StrOutputParser\n",
        "from langchain_openai import ChatOpenAI\n",
        "\n",
        "chat_model = ChatOpenAI(model=\"gpt-3.5-turbo\")\n",
        "\n",
        "rag_chain = (\n",
        "    {\"question\": itemgetter(\"question\"), \"context\": itemgetter(\"question\") | qdrant_retriever}\n",
        "    | RunnablePassthrough().assign(context=itemgetter(\"context\"))\n",
        "    | {\"response\":rag_prompt | chat_model | StrOutputParser(), \"context\": itemgetter(\"context\")}\n",
        ")"
      ],
      "metadata": {
        "id": "ptOBMYQiAHKG"
      },
      "execution_count": 85,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Query the Meta 10-K Form"
      ],
      "metadata": {
        "id": "IYpC7OUInDBU"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Great, time to query the Form!"
      ],
      "metadata": {
        "id": "tAhPVqoOnMFv"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# query the rag_chain\n",
        "query1 = \"What was the total value of 'Cash and cash equivalents' as of December 31, 2023?\"\n",
        "response = rag_chain.invoke({\"question\": query1})\n",
        "print(response['response'])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "83d6a53e-12d0-480c-be60-20cff724ca0e",
        "id": "tO61pDX012aN"
      },
      "execution_count": 87,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The total value of 'Cash and cash equivalents' as of December 31, 2023, was $41,862 million.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "for context in response['context']:\n",
        "    print('======== CONTEXT ========')\n",
        "    print(context.page_content)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "v03JzLwwjIBm",
        "outputId": "4ad79d63-8817-4433-e9f6-f3969d8d1207"
      },
      "execution_count": 88,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "======== CONTEXT ========\n",
            "Fair Value Measurement at Reporting Date Using\n",
            "|Description|December 31, 2023|Quoted Prices in Active Markets for Identical Assets (Level 1)|Significant Observable Inputs (Level 2)|Significant Unobservable Inputs (Level 3)|\n",
            "|---|---|---|---|---|\n",
            "|Cash|$6,265| | | |\n",
            "|Cash equivalents: Money market funds|$32,910|$32,910| | |\n",
            "|Cash equivalents: U.S. government and agency securities|$2,206|$2,206| | |\n",
            "|Cash equivalents: Time deposits|$261| |$261| |\n",
            "|Cash equivalents: Corporate debt securities|$220| |$220| |\n",
            "|Total cash and cash equivalents|$41,862|$35,116|$481| |\n",
            "|Marketable securities: U.S. government securities|$8,439|$8,439| | |\n",
            "|Marketable securities: U.S. government agency securities|$3,498|$3,498| | |\n",
            "|Marketable securities: Corporate debt securities|$11,604| |$11,604| |\n",
            "|Total marketable securities|$23,541|$11,937|$11,604| |\n",
            "|Restricted cash equivalents|$857|$857| | |\n",
            "|Other assets|$101| | |$101|\n",
            "|Total|$66,361|$47,910|$12,085|$101|\n",
            "\n",
            "107\n",
            "\n",
            "Meta Platforms, Inc. - Annual Report\n",
            "\n",
            "Table of Contents\n",
            "\n",
            "Fair Value Measurement at Reporting Date Using\n",
            "\n",
            "Description December 31, 2022 Quoted Prices in Active Markets for Identical Assets (Level 1) Significant Other Observable Inputs (Level 2) Significant Unobservable Inputs (Level 3) Cash $6,176 Cash equivalents: Money market funds $8,305 $8,305 Cash equivalents: U.S. government and agency securities $16 $16 Cash equivalents: Time deposits $156 $156 Cash equivalents: Corporate debt securities $28 $28 Total cash and cash equivalents $14,681 $8,321 $184 Marketable securities: U.S. government securities $8,708 $8,708 Marketable securities: U.S. government agency securities $4,989 $4,989 Marketable securities: Corporate debt securities $12,335 $12,335 Marketable securities: Marketable equity securities $25 $25 Total marketable securities $26,057 $13,722 $12,335 Restricted cash equivalents $583 $583 Other assets $157 $157 Total $41,478 $22,626 $12,519 $157\n",
            "\n",
            "Unrealized Losses\n",
            "\n",
            "The following tables summarize our available-for-sale marketable debt securities and cash equivalents with unrealized losses as of December 31, 2023 and 2022, aggregated by major security type and the length of time that individual securities have been in a continuous loss position (in millions):\n",
            "\n",
            "December 31, 2023\n",
            "\n",
            "Less than 12 months 12 months or greater Total U.S. government securities $336 $7,041 $7,377 U.S. government agency securities $71 $3,225 $3,296 Corporate debt securities $647 $10,125 $10,772 Total $1,054 $20,391 $21,445\n",
            "======== CONTEXT ========\n",
            "Fair Value Measurement at Reporting Date Using\n",
            "|Description|December 31, 2023|Quoted Prices in Active Markets for Identical Assets (Level 1)|Significant Observable Inputs (Level 2)|Significant Unobservable Inputs (Level 3)|\n",
            "|---|---|---|---|---|\n",
            "|Cash|$6,265| | | |\n",
            "|Cash equivalents: Money market funds|$32,910|$32,910| | |\n",
            "|Cash equivalents: U.S. government and agency securities|$2,206|$2,206| | |\n",
            "|Cash equivalents: Time deposits|$261| |$261| |\n",
            "|Cash equivalents: Corporate debt securities|$220| |$220| |\n",
            "|Total cash and cash equivalents|$41,862|$35,116|$481| |\n",
            "|Marketable securities: U.S. government securities|$8,439|$8,439| | |\n",
            "|Marketable securities: U.S. government agency securities|$3,498|$3,498| | |\n",
            "|Marketable securities: Corporate debt securities|$11,604| |$11,604| |\n",
            "|Total marketable securities|$23,541|$11,937|$11,604| |\n",
            "|Restricted cash equivalents|$857|$857| | |\n",
            "|Other assets|$101| | |$101|\n",
            "|Total|$66,361|$47,910|$12,085|$101|\n",
            "\n",
            "107\n",
            "\n",
            "Meta Platforms, Inc. - Annual Report\n",
            "\n",
            "Table of Contents\n",
            "\n",
            "Fair Value Measurement at Reporting Date Using\n",
            "\n",
            "Description December 31, 2022 Quoted Prices in Active Markets for Identical Assets (Level 1) Significant Other Observable Inputs (Level 2) Significant Unobservable Inputs (Level 3) Cash $6,176 Cash equivalents: Money market funds $8,305 $8,305 Cash equivalents: U.S. government and agency securities $16 $16 Cash equivalents: Time deposits $156 $156 Cash equivalents: Corporate debt securities $28 $28 Total cash and cash equivalents $14,681 $8,321 $184 Marketable securities: U.S. government securities $8,708 $8,708 Marketable securities: U.S. government agency securities $4,989 $4,989 Marketable securities: Corporate debt securities $12,335 $12,335 Marketable securities: Marketable equity securities $25 $25 Total marketable securities $26,057 $13,722 $12,335 Restricted cash equivalents $583 $583 Other assets $157 $157 Total $41,478 $22,626 $12,519 $157\n",
            "\n",
            "Unrealized Losses\n",
            "\n",
            "The following tables summarize our available-for-sale marketable debt securities and cash equivalents with unrealized losses as of December 31, 2023 and 2022, aggregated by major security type and the length of time that individual securities have been in a continuous loss position (in millions):\n",
            "\n",
            "December 31, 2023\n",
            "\n",
            "Less than 12 months 12 months or greater Total U.S. government securities $336 $7,041 $7,377 U.S. government agency securities $71 $3,225 $3,296 Corporate debt securities $647 $10,125 $10,772 Total $1,054 $20,391 $21,445\n",
            "======== CONTEXT ========\n",
            "The following tables summarize our assets measured at fair value on a recurring basis and the classification by level of input within the fair value hierarchy (in millions):\n",
            "\n",
            "Fair Value Measurement at Reporting Date Using\n",
            "|Description|December 31, 2023|Quoted Prices in Active Markets for Identical Assets (Level 1)|Significant Other Observable Inputs (Level 2)|Significant Unobservable Inputs (Level 3)|\n",
            "|---|---|---|---|---|\n",
            "|Cash|$6,265| | | |\n",
            "|Cash equivalents: Money market funds|$32,910|$32,910| | |\n",
            "|Cash equivalents: U.S. government and agency securities|$2,206|$2,206| | |\n",
            "|Cash equivalents: Time deposits|$261| |$261| |\n",
            "|Cash equivalents: Corporate debt securities|$220| |$220| |\n",
            "|Total cash and cash equivalents|$41,862|$35,116|$481| |\n",
            "|Marketable securities: U.S. government securities|$8,439|$8,439| | |\n",
            "|Marketable securities: U.S. government agency securities|$3,498|$3,498| | |\n",
            "|Marketable securities: Corporate debt securities|$11,604| |$11,604| |\n",
            "|Total marketable securities|$23,541|$11,937|$11,604| |\n",
            "|Restricted cash equivalents|$857|$857| | |\n",
            "|Other assets|$101| | |$101|\n",
            "|Total|$66,361|$47,910|$12,085|$101|\n",
            "\n",
            "107\n",
            "\n",
            "Meta Platforms, Inc. - Annual Report\n",
            "\n",
            "Table of Contents\n",
            "\n",
            "Fair Value Measurement at Reporting Date Using\n",
            "\n",
            "Description December 31, 2022 Quoted Prices in Active Markets for Identical Assets (Level 1) Significant Other Observable Inputs (Level 2) Significant Unobservable Inputs (Level 3) Cash $6,176 Cash equivalents: Money market funds $8,305 $8,305 Cash equivalents: U.S. government and agency securities $16 $16 Cash equivalents: Time deposits $156 $156 Cash equivalents: Corporate debt securities $28 $28 Total cash and cash equivalents $14,681 $8,321 $184 Marketable securities: U.S. government securities $8,708 $8,708 Marketable securities: U.S. government agency securities $4,989 $4,989 Marketable securities: Corporate debt securities $12,335 $12,335 Marketable securities: Marketable equity securities $25 $25 Total marketable securities $26,057 $13,722 $12,335 Restricted cash equivalents $583 $583 Other assets $157 $157 Total $41,478 $22,626 $12,519 $157\n",
            "\n",
            "Unrealized Losses\n",
            "\n",
            "The following tables summarize our available-for-sale marketable debt securities and cash equivalents with unrealized losses as of December 31, 2023 and 2022, aggregated by major security type and the length of time that individual securities have been in a continuous loss position (in millions):\n",
            "\n",
            "December 31, 2023\n",
            "======== CONTEXT ========\n",
            "included in other assets 866 621 115 Total cash, cash equivalents, and restricted cash $42,827 $15,596 $16,865\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# query the rag_chain\n",
        "query2 = \"Who are Meta's 'Directors' (i.e., members of the Board of Directors)?\"\n",
        "response = rag_chain.invoke({\"question\": query2})\n",
        "print(response['response'])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "db77cf3c-0e28-4cce-b0c6-b1103845671d",
        "id": "l5KLh5Nh12aN"
      },
      "execution_count": 89,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The Directors of Meta Platforms, Inc. listed in the document are:\n",
            "\n",
            "- Mark Zuckerberg\n",
            "- Susan Li\n",
            "- Aaron Anderson\n",
            "- Peggy Alford\n",
            "- Marc L. Andreessen\n",
            "- Andrew W. Houston\n",
            "- Nancy Killefer\n",
            "- Robert M. Kimmitt\n",
            "- Sheryl K. Sandberg\n",
            "- Tracey T. Travis\n",
            "- Tony Xu\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "for context in response['context']:\n",
        "    print('======== CONTEXT ========')\n",
        "    print(context.page_content)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "630f0d9f-49e5-4d29-d768-a097e24694f9",
        "id": "yzrKRr3tjpJi"
      },
      "execution_count": 90,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "======== CONTEXT ========\n",
            "10\n",
            "\n",
            "Meta Platforms, Inc. - Annual Report\n",
            "\n",
            "Table of Contents\n",
            "\n",
            "Table of contents content goes here...\n",
            "\n",
            "Signatory Title Date Mark Zuckerberg Chief Executive Officer March 1, 2024 Sheryl Sandberg Chief Operating Officer March 1, 2024 David Wehner Chief Financial Officer March 1, 2024 --- # Meta Platforms, Inc. Annual Report\n",
            "\n",
            "Meta Platforms, Inc. Annual Report\n",
            "\n",
            "Signatures\n",
            "\n",
            "Name Title Date [Signature Name 1] [Title 1] [Date 1] [Signature Name 2] [Title 2] [Date 2] [Signature Name 3] [Title 3] [Date 3] --- # Meta Platforms, Inc. Annual Report\n",
            "\n",
            "Table of Contents\n",
            "\n",
            "Compensation, Benefits, Health, and Well-being\n",
            "\n",
            "We offer competitive compensation to attract and retain the best people, and we help care for our people so they can focus on our mission. Our employees' total compensation package includes market-competitive salary, bonuses or sales incentives, and equity. We generally offer full-time employees equity at the time of hire and through annual equity grants because we want them to be owners of the company and committed to our long-term success. We have conducted pay equity analyses for many years, and continue to be committed to pay equity. For example, in July 2023, we announced that our analyses confirm that we continue to have pay equity across genders globally and by race in the United States for people in similar jobs, accounting for factors such as location, role, and level.\n",
            "\n",
            "Through Life@ Meta, our holistic approach to benefits, we continue to provide our employees and their dependents with resources to help them thrive. We offer a wide range of benefits across areas such as health, family, finance, community, and time away, including family building benefits, family care resources, retirement savings plans, access to legal services, Meta Resource Groups to build community at Meta, and health and well-being benefits.\n",
            "\n",
            "Our health and well-being programs are designed to give employees a choice of flexible benefits to help them reach their personal well-being goals. Our programs are tailored to help boost employee physical and mental health, create financial peace of mind, provide support for families, and help employees build a strong community. Programs are designed and funded to support needs like autism care, cancer care, transgender services, holistic well-being, including mental health programs and retirement savings, which represent a few of the ways we support our employees and their dependents.\n",
            "\n",
            "Diverse and Inclusive Workplace\n",
            "======== CONTEXT ========\n",
            "Meta Platforms, Inc. - List of Subsidiaries\n",
            "\n",
            "List of Subsidiaries - Meta Platforms, Inc.\n",
            "\n",
            "Subsidiary Name Incorporation Cassin Networks ApS (Denmark) Edge Network Services Limited (Ireland) Facebook Circularity, LLC (Delaware) Facebook Holdings, LLC (Delaware) Facebook India Online Services Private Limited (India) Facebook Operations, LLC (Delaware) Facebook Procurement LLC (Delaware) Facebook Serviços Online Do Brasil Ltda. (Brazil) Facebook UK Limited (United Kingdom) FCL Tech Limited (Ireland) Goldframe LLC (Delaware) Greater Kudu LLC (Delaware) Hibiscus Properties, LLC (Delaware) Instagram, LLC (Delaware) Malkoha Pte. Ltd. (Singapore) Meta Payments Inc. (Florida) Meta Platforms Ireland Limited (Ireland) Meta Platforms Technologies, LLC (Delaware) Morning Hornet LLC (Delaware) Pinnacle Sweden AB (Sweden) Raven Northbrook LLC (Delaware) Redale LLC (Delaware) Runways Information Services Limited (Ireland) Scout Development, LLC (Delaware) Siculus, Inc. (Delaware) Sidecat LLC (Delaware) Stadion LLC (Delaware) Starbelt LLC (Delaware) Vitesse, LLC (Delaware) WhatsApp LLC (Delaware) Winner LLC (Delaware) Woolhawk LLC (Delaware) --- ```markdown Firm Name Ernst & Young LLP ------------------ ------------------- Location San Mateo, California Date February 1, 2024 ``` --- Name: Mark Zuckerberg --- --- Title: Board Chair and Chief Executive Officer (Principal Executive Officer) Date: February 1, 2024 --- Date: February 1, 2024 --- --- /s/ SUSAN LI Susan Li Susan Li Chief Financial Officer (Principal Financial Officer) --- Name: Mark Zuckerberg --- --- Title: Board Chair and Chief Executive Officer Date: February 1, 2024 --- Date: February 1, 2024 --- ---    /s/ SUSAN LI Susan Li Chief Financial Officer (Principal Financial Officer) --- # Meta Platforms, Inc. - Compensation Recoupment Policy\n",
            "\n",
            "META PLATFORMS, INC. COMPENSATION RECOUPMENT POLICY\n",
            "======== CONTEXT ========\n",
            "Chief Executive Officer\n",
            "Not specified  \n",
            "Meta Platforms, Inc. - Annual Report  \n",
            "Table of Contents  \n",
            "Exhibit Number Exhibit Description Form File No. Exhibit Filing Date Herewith 32.2# Certification of Susan Li, Chief Financial Officer, pursuant to 18 U.S.C. Section 1350, as adopted pursuant to Section 906 of the Sarbanes-Oxley Act of 2002. X 97.1 Compensation Recoupment Policy. X 101.INS Inline XBRL Instance Document (the instance document does not appear in the Interactive Data File because its XBRL tags are embedded within the Inline XBRL document). X 101.SCH Inline XBRL Taxonomy Extension Schema Document. X 101.CAL Inline XBRL Taxonomy Extension Calculation Linkbase Document. X 101.DEF Inline XBRL Taxonomy Extension Definition Linkbase Document. X 101.LAB Inline XBRL Taxonomy Extension Labels Linkbase Document. X 101.PRE Inline XBRL Taxonomy Extension Presentation Linkbase Document. X 104 Cover Page Interactive Data File (formatted as inline XBRL and contained in Exhibit 101). X  \n",
            "Indicates a management contract or compensatory plan.  \n",
            "This certification is deemed not filed for purposes of Section 18 of the Securities Exchange Act of 1934, as amended (Exchange Act), or otherwise subject to the liability of that section, nor shall it be deemed incorporated by reference into any filing under the Securities Act of 1933, as amended, or the Exchange Act.  \n",
            "Item 16. Form 10-K Summary  \n",
            "None.  \n",
            "130  \n",
            "Meta Platforms, Inc. - Signatures  \n",
            "Date Signatory Title February 1, 2024 Susan Li Chief Financial Officer --- # Meta Platforms, Inc. - Signatures  \n",
            "Signature\n",
            "Title\n",
            "Date  \n",
            "/s/ Mark Zuckerberg\n",
            "Board Chair and Chief Executive Officer\n",
            "February 1, 2024  \n",
            "/s/ Susan Li\n",
            "Chief Financial Officer\n",
            "February 1, 2024  \n",
            "/S/ Aaron Anderson\n",
            "Chief Accounting Officer\n",
            "February 1, 2024  \n",
            "/s/ Peggy Alford\n",
            "Director\n",
            "February 1, 2024  \n",
            "/s/ Marc L. Andreessen\n",
            "Director\n",
            "February 1, 2024  \n",
            "/s/ Andrew W. Houston\n",
            "Director\n",
            "February 1, 2024  \n",
            "/s/ Nancy Killefer\n",
            "Director\n",
            "February 1, 2024  \n",
            "/s/ Robert M. Kimmitt\n",
            "Director\n",
            "February 1, 2024  \n",
            "/s/ Sheryl K. Sandberg\n",
            "Director\n",
            "February 1, 2024  \n",
            "/s/ Tracey T. Travis\n",
            "Director\n",
            "February 1, 2024  \n",
            "/s/ Tony Xu\n",
            "Director\n",
            "February 1, 2024  \n",
            "Meta Platforms, Inc. - Description of Capital Stock  \n",
            "DESCRIPTION OF CAPITAL STOCK  \n",
            "The following description of capital stock of Meta Platforms, Inc. (the “company,” “we,” “us” and “our”) summarizes certain provisions of our amended\n",
            "======== CONTEXT ========\n",
            "Signatures  \n",
            "Name Title Date [Signature] Mark Zuckerberg Chief Executive Officer [Signature] David M. Wehner Chief Financial Officer --- # Meta Platforms, Inc. - List of Subsidiaries  \n",
            "List of Subsidiaries - Meta Platforms, Inc.  \n",
            "Subsidiary Name\n",
            "Incorporation  \n",
            "Cassin Networks ApS (Denmark)  \n",
            "Edge Network Services Limited (Ireland)  \n",
            "Facebook Circularity, LLC (Delaware)  \n",
            "Facebook Holdings, LLC (Delaware)  \n",
            "Facebook India Online Services Private Limited (India)  \n",
            "Facebook Operations, LLC (Delaware)  \n",
            "Facebook Procurement LLC (Delaware)  \n",
            "Facebook Serviços Online Do Brasil Ltda. (Brazil)  \n",
            "Facebook UK Limited (United Kingdom)  \n",
            "FCL Tech Limited (Ireland)  \n",
            "Goldframe LLC (Delaware)  \n",
            "Greater Kudu LLC (Delaware)  \n",
            "Hibiscus Properties, LLC (Delaware)  \n",
            "Instagram, LLC (Delaware)  \n",
            "Malkoha Pte. Ltd. (Singapore)  \n",
            "Meta Payments Inc. (Florida)  \n",
            "Meta Platforms Ireland Limited (Ireland)  \n",
            "Meta Platforms Technologies, LLC (Delaware)  \n",
            "Morning Hornet LLC (Delaware)  \n",
            "Pinnacle Sweden AB (Sweden)  \n",
            "Raven Northbrook LLC (Delaware)  \n",
            "Redale LLC (Delaware)  \n",
            "Runways Information Services Limited (Ireland)  \n",
            "Scout Development, LLC (Delaware)  \n",
            "Siculus, Inc. (Delaware)  \n",
            "Sidecat LLC (Delaware)  \n",
            "Stadion LLC (Delaware)  \n",
            "Starbelt LLC (Delaware)  \n",
            "Vitesse, LLC (Delaware)  \n",
            "WhatsApp LLC (Delaware)  \n",
            "Winner LLC (Delaware)  \n",
            "Woolhawk LLC (Delaware)  \n",
            "Meta Platforms, Inc. - Consent of Independent Registered Public Accounting Firm  \n",
            "Registration Statement\n",
            "Description  \n",
            "Form S-8 No. 333-270184\n",
            "2012 Equity Incentive Plan of Meta Platforms, Inc.  \n",
            "Form S-8 No. 333-262508\n",
            "2012 Equity Incentive Plan of Meta Platforms, Inc.  \n",
            "Form S-8 No. 333-252518\n",
            "2012 Equity Incentive Plan of Meta Platforms, Inc.  \n",
            "Form S-8 No. 333-236161\n",
            "2012 Equity Incentive Plan of Meta Platforms, Inc.  \n",
            "Form S-8 No. 333-229457\n",
            "2012 Equity Incentive Plan of Meta Platforms, Inc.  \n",
            "Form S-8 No. 333-222823\n",
            "2012 Equity Incentive Plan of Meta Platforms, Inc.  \n",
            "Form S-8 No. 333-186402\n",
            "2012 Equity Incentive Plan of Meta Platforms, Inc.  \n",
            "Form S-8 No. 333-181566\n",
            "2005 Officers’ Stock Plan, 2005 Stock Plan, and 2012 Equity Incentive Plan of Meta Platforms, Inc.  \n",
            "Form S-3 No. 333-271535\n",
            "Meta Platforms, Inc.  \n",
            "Consent of Ernst & Young LLP, San Mateo, California, dated February 1, 2024, regarding the consolidated financial statements and internal control over financial reporting of Meta Platforms, Inc. for the year ended December 31, 2023.  \n",
            "Meta Platforms, Inc. - Annual Report\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "Pog6YfXRqs84"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# query the rag_chain\n",
        "response = rag_chain.invoke({\"question\": \"What's the par value of Meta's Class A common stock?\"})\n",
        "print(response['response'])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b64b17fb-c707-4628-e907-04a8425285fc",
        "id": "QEUGDMqnOBYL"
      },
      "execution_count": 91,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The par value of Meta's Class A common stock is $0.000006 per share.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# query the rag_chain\n",
        "response = rag_chain.invoke({\"question\": \"What is Meta's dividend policy?\"})\n",
        "print(response['response'])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1d19b246-5a07-49c4-ede7-11fe92e335ab",
        "id": "2pIq--IfOToh"
      },
      "execution_count": 95,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Meta's dividend policy states that prior to 2024, the company had never declared or paid any cash dividend on their common stock. However, on February 1, 2024, they announced the initiation of their first ever cash dividend program. This program includes a cash dividend of $0.50 per share of Class A common stock and Class B common stock, equivalent to $2.00 per share on an annual basis. The first cash dividend was scheduled to be paid on March 26, 2024 to all holders of record of common stock at the close of business on February 22, 2024. The payment of future cash dividends is subject to future declaration by their board of directors, based on various factors including capital availability, market conditions, laws, and the best interests of stockholders.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# query the rag_chain\n",
        "response = rag_chain.invoke({\"question\": \"What is Meta's current net worth?\"})\n",
        "print(response['response'])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "62ddea20-e2e7-4fab-b4f0-d147dbad18f7",
        "id": "3BhBr0GdO8KW"
      },
      "execution_count": 96,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Meta's current net worth, based on the information provided in the annual report, is $229,623 million as of December 31, 2023.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "B_s-Pep5OOnS"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}