{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "view-in-github"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/towardsai/ai-tutor-rag-system/blob/main/notebooks/Crawl_a_Website.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4CW8ux1RSdem",
        "outputId": "155feab4-8ae6-43da-a07f-8a1f4b677c2b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m211.1/211.1 kB\u001b[0m \u001b[31m4.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m81.3/81.3 kB\u001b[0m \u001b[31m8.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m97.6/97.6 kB\u001b[0m \u001b[31m9.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.4/7.4 MB\u001b[0m \u001b[31m43.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Building wheel for tinysegmenter (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Building wheel for feedfinder2 (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Building wheel for jieba3k (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Building wheel for sgmllib3k (setup.py) ... \u001b[?25l\u001b[?25hdone\n"
          ]
        }
      ],
      "source": [
        "!pip install -q llama-index==0.10.57 llama-index-llms-gemini==0.1.11 openai==1.37.0 google-generativeai==0.5.4 newspaper3k==0.2.8"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wxDPsVXSAj6_"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "# Set the following API Keys in the Python environment. Will be used later.\n",
        "os.environ[\"OPENAI_API_KEY\"] = \"[OPENAI_API_KEY]\"\n",
        "USESCRAPER_API_KEY = \"[USESCRAPER_API_KEY]\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VSc7-1mljmrp"
      },
      "source": [
        "There are two primary methods for extracting webpage content. The first method involves having a list of URLs; one can iterate through this list to retrieve the content of each page. The second method, web crawling, requires using a script or service to extract page URLs from a sitemap or manually following links on the page to access all the content. Initially, we will explore web scraping techniques before discussing how to use a service like usescraper.com to perform web crawling.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D3r2tYHgeIK9"
      },
      "source": [
        "# 1. Scraping using `newspaper` Library\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "it43ZQf8jatw"
      },
      "source": [
        "## Define URLs\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "x74PqfQ7eIzD"
      },
      "outputs": [],
      "source": [
        "urls = [\n",
        "    \"https://docs.llamaindex.ai/en/stable/understanding\",\n",
        "    \"https://docs.llamaindex.ai/en/stable/understanding/using_llms/using_llms/\",\n",
        "    \"https://docs.llamaindex.ai/en/stable/understanding/indexing/indexing/\",\n",
        "    \"https://docs.llamaindex.ai/en/stable/understanding/querying/querying/\",\n",
        "]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tgxfpfSsjcMC"
      },
      "source": [
        "## Get Page Contents\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Q6Xs1OhUfVQV"
      },
      "outputs": [],
      "source": [
        "import newspaper\n",
        "\n",
        "pages_content = []\n",
        "\n",
        "# Retrieve the Content\n",
        "for url in urls:\n",
        "    try:\n",
        "        article = newspaper.Article(url)\n",
        "        article.download()\n",
        "        article.parse()\n",
        "        if len(article.text) > 0:\n",
        "            pages_content.append(\n",
        "                {\"url\": url, \"title\": article.title, \"text\": article.text}\n",
        "            )\n",
        "    except:\n",
        "        continue"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3cNdJNi2g1ly",
        "outputId": "f5184c15-6b55-47ee-98ee-646a06290a4c"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'url': 'https://docs.llamaindex.ai/en/stable/understanding',\n",
              " 'title': 'Building an LLM Application',\n",
              " 'text': \"Building an LLM application#\\n\\nWelcome to the beginning of Understanding LlamaIndex. This is a series of short, bite-sized tutorials on every stage of building an LLM application to get you acquainted with how to use LlamaIndex before diving into more advanced and subtle strategies. If you're an experienced programmer new to LlamaIndex, this is the place to start.\\n\\nKey steps in building an LLM application#\\n\\nTip If you've already read our high-level concepts page you'll recognize several of these steps.\\n\\nThere are a series of key steps involved in building any LLM-powered application, whether it's answering questions about your data, creating a chatbot, or an autonomous agent. Throughout our documentation, you'll notice sections are arranged roughly in the order you'll perform these steps while building your app. You'll learn about:\\n\\nUsing LLMs : whether it's OpenAI or any number of hosted LLMs or a locally-run model of your own, LLMs are used at every step of the way, from indexing and storing to querying and parsing your data. LlamaIndex comes with a huge number of reliable, tested prompts and we'll also show you how to customize your own.\\n\\nLoading : getting your data from wherever it lives, whether that's unstructured text, PDFs, databases, or APIs to other applications. LlamaIndex has hundreds of connectors to every data source over at LlamaHub.\\n\\nIndexing : once you've got your data there are an infinite number of ways to structure access to that data to ensure your applications is always working with the most relevant data. LlamaIndex has a huge number of these strategies built-in and can help you select the best ones.\\n\\nStoring : you will probably find it more efficient to store your data in indexed form, or pre-processed summaries provided by an LLM, often in a specialized database known as a Vector Store (see below). You can also store your indexes, metadata and more.\\n\\nQuerying : every indexing strategy has a corresponding querying strategy and there are lots of ways to improve the relevance, speed and accuracy of what you retrieve and what the LLM does with it before returning it to you, including turning it into structured responses such as an API.\\n\\nPutting it all together : whether you are building question & answering, chatbots, an API, or an autonomous agent, we show you how to get your application into production.\\n\\nTracing and debugging : also called observability , it's especially important with LLM applications to be able to look into the inner workings of what's going on to help you debug problems and spot places to improve.\\n\\nEvaluating: every strategy has pros and cons and a key part of building, shipping and evolving your application is evaluating whether your change has improved your application in terms of accuracy, performance, clarity, cost and more. Reliably evaluating your changes is a crucial part of LLM application development.\\n\\nReady to dive in? Head to using LLMs.\"}"
            ]
          },
          "execution_count": 57,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "pages_content[0]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WleP60A3gkQM",
        "outputId": "8c79ab53-e47b-4227-eb6f-0286b8ba2d15"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "5"
            ]
          },
          "execution_count": 38,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "len(pages_content)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i5mCiRfGjfNx"
      },
      "source": [
        "## Convert to Document\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TOJ3K-CBfVDR"
      },
      "outputs": [],
      "source": [
        "from llama_index.core.schema import Document\n",
        "\n",
        "# Convert the chunks to Document objects so the LlamaIndex framework can process them.\n",
        "documents = [\n",
        "    Document(text=row[\"text\"], metadata={\"title\": row[\"title\"], \"url\": row[\"url\"]})\n",
        "    for row in pages_content\n",
        "]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CkjEyEmkJevT"
      },
      "source": [
        "# 2. Submit the Crawler Job\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tYpchBo5-brp",
        "outputId": "927f84c5-c13a-408c-8802-df90bc05c733"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{'org': '581', 'id': '7YE3T8VSPJVSCYE6EDQ90DJNFT', 'urls': ['https://docs.llamaindex.ai/en/stable/understanding/'], 'exclude_globs': [], 'exclude_elements': 'nav, header, footer, script, style, noscript, svg, [role=\"alert\"], [role=\"banner\"], [role=\"dialog\"], [role=\"alertdialog\"], [role=\"region\"][aria-label*=\"skip\" i], [aria-modal=\"true\"]', 'output_format': 'markdown', 'output_expiry': 604800, 'min_length': 50, 'page_limit': 10000, 'force_crawling_mode': 'link', 'block_resources': True, 'include_linked_files': False, 'createdAt': 1713883978029, 'status': 'starting', 'use_browser': True, 'sitemapPageCount': 0, 'notices': []}\n"
          ]
        }
      ],
      "source": [
        "import requests\n",
        "import json\n",
        "\n",
        "payload = {\n",
        "    \"urls\": [\n",
        "        \"https://docs.llamaindex.ai/en/stable/understanding/\"\n",
        "    ],  # list of urls to crawl\n",
        "    \"output_format\": \"markdown\",  # text, html, markdown\n",
        "    \"output_expiry\": 604800,  # Automatically delete after X seconds\n",
        "    \"min_length\": 50,  # Skip pages with less than X characters\n",
        "    \"page_limit\": 10000,  # Maximum number of pages to crawl\n",
        "    \"force_crawling_mode\": \"link\",  # \"link\" follows links in the page reccursively, or \"sitemap\" to find pages from website's sitemap\n",
        "    \"block_resources\": True,  # skip loading images, stylesheets, or scripts\n",
        "    \"include_linked_files\": False,  # include files (PDF, text, ...) in output\n",
        "}\n",
        "headers = {\n",
        "    \"Authorization\": \"Bearer \" + USESCRAPER_API_KEY,\n",
        "    \"Content-Type\": \"application/json\",\n",
        "}\n",
        "\n",
        "response = requests.request(\n",
        "    \"POST\", \"https://api.usescraper.com/crawler/jobs\", json=payload, headers=headers\n",
        ")\n",
        "\n",
        "response = json.loads(response.text)\n",
        "\n",
        "print(response)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nx_4MjHxJgxh"
      },
      "source": [
        "## Get the Status\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZLJ0BUR8c1a8",
        "outputId": "cfd3aee9-68bf-4171-9340-abe2d03fa5ac"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "running\n",
            "{'scraped': 9, 'discarded': 0, 'failed': 0}\n"
          ]
        }
      ],
      "source": [
        "url = \"https://api.usescraper.com/crawler/jobs/{}\".format(response[\"id\"])\n",
        "\n",
        "status_res = requests.request(\"GET\", url, headers=headers)\n",
        "\n",
        "status_res = json.loads(status_res.text)\n",
        "\n",
        "print(status_res[\"status\"])\n",
        "print(status_res[\"progress\"])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vHcRJIDsJh2i"
      },
      "source": [
        "## Get the Data\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "J4dUn4cmGGab"
      },
      "outputs": [],
      "source": [
        "url = \"https://api.usescraper.com/crawler/jobs/{}/data\".format(response[\"id\"])\n",
        "\n",
        "data_res = requests.request(\"GET\", url, headers=headers)\n",
        "\n",
        "data_res = json.loads(data_res.text)\n",
        "\n",
        "print(data_res)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "F8VEQvJkITLJ",
        "outputId": "b54ec108-7221-4230-8b61-d0a4be503a66"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "URL: https://docs.llamaindex.ai/en/stable/understanding/putting_it_all_together/graphs/\n",
            "Title: Knowledge Graphs - LlamaIndex\n",
            "Content:  \n",
            "[ Skip to content ](https://docs.llamaindex.ai/en/stable/understanding/putting_it_all_together/graphs/#knowledge-graphs)\n",
            "#Knowledge Graphs[#](https://docs.llamaindex.ai/en/stable/understanding/putting_it_all_together/graphs/#knowledge-graphs)\n",
            "LlamaIndex contains some fantastic guides for building with knowledge graphs.\n",
            "\n",
            "Check out the end-to-end tutorials/workshops below. Also check out our [knowledge graph query engine guides](https://docs.llamaindex.ai/en/stable/module_guides/deploying/query_ ...\n"
          ]
        }
      ],
      "source": [
        "print(\"URL:\", data_res[\"data\"][0][\"meta\"][\"url\"])\n",
        "print(\"Title:\", data_res[\"data\"][0][\"meta\"][\"meta\"][\"title\"])\n",
        "print(\"Content:\", data_res[\"data\"][0][\"text\"][0:500], \"...\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rt2nyuLhSYLR"
      },
      "source": [
        "## Convert to Document\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YEieGzSFSXas"
      },
      "outputs": [],
      "source": [
        "from llama_index.core.schema import Document\n",
        "\n",
        "# Convert the chunks to Document objects so the LlamaIndex framework can process them.\n",
        "documents = [\n",
        "    Document(\n",
        "        text=row[\"text\"],\n",
        "        metadata={\"title\": row[\"meta\"][\"meta\"][\"title\"], \"url\": row[\"meta\"][\"url\"]},\n",
        "    )\n",
        "    for row in data_res[\"data\"]\n",
        "]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vqbJG5a1i3Jo"
      },
      "source": [
        "# Create RAG Pipeline\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wxmiQDv3SXV6"
      },
      "outputs": [],
      "source": [
        "from llama_index.llms.gemini import Gemini\n",
        "\n",
        "llm = Gemini(model=\"models/gemini-1.5-flash\", temperature=1, max_tokens=512)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tCVhv4OkSXTV"
      },
      "outputs": [],
      "source": [
        "from llama_index.embeddings.openai import OpenAIEmbedding\n",
        "\n",
        "embed_model = OpenAIEmbedding(model=\"text-embedding-3-large\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "quwJI61dNVr-"
      },
      "outputs": [],
      "source": [
        "from llama_index.core.node_parser import SentenceSplitter\n",
        "\n",
        "text_splitter = SentenceSplitter(chunk_size=512, chunk_overlap=30)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6KpeCRMBUgup"
      },
      "outputs": [],
      "source": [
        "from llama_index.core import Settings\n",
        "\n",
        "Settings.llm = llm\n",
        "Settings.embed_model = embed_model\n",
        "Settings.text_splitter = text_splitter"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "nWTBidwoZSO0"
      },
      "outputs": [],
      "source": [
        "from llama_index.core import VectorStoreIndex\n",
        "\n",
        "index = VectorStoreIndex.from_documents(documents)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "RUuJO0IIYSeU"
      },
      "outputs": [],
      "source": [
        "query_engine = index.as_query_engine()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6_s2LkH6YX1V"
      },
      "outputs": [],
      "source": [
        "res = query_engine.query(\"What is a query engine?\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 71
        },
        "id": "02zdJNqIZKep",
        "outputId": "76340610-0d98-4fd0-d237-ddb9f1752391"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'A query engine is a fundamental component used in querying processes. It is responsible for retrieving the most relevant documents from an index based on a query, postprocessing the retrieved nodes if needed, and then synthesizing a response by combining the query, relevant data, and prompt to be sent to the language model for generating an answer.'"
            ]
          },
          "execution_count": 28,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "res.response"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PuCcgP0nZSIl",
        "outputId": "e136cdbb-2ee4-4dfb-f532-f6c9365e519e"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Node ID\t 081b6c8c-d9ea-4476-bac0-1008facd3db8\n",
            "Title\t Querying - LlamaIndex\n",
            "URL\t https://docs.llamaindex.ai/en/stable/understanding/querying/querying/\n",
            "Score\t 0.46212738505767387\n",
            "-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_\n",
            "Node ID\t 3786c195-c5de-4bba-98b6-996031349a88\n",
            "Title\t Querying - LlamaIndex\n",
            "URL\t https://docs.llamaindex.ai/en/stable/understanding/querying/querying/\n",
            "Score\t 0.43141762602042416\n",
            "-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_\n"
          ]
        }
      ],
      "source": [
        "# Show the retrieved nodes\n",
        "for src in res.source_nodes:\n",
        "    print(\"Node ID\\t\", src.node_id)\n",
        "    print(\"Title\\t\", src.metadata[\"title\"])\n",
        "    print(\"URL\\t\", src.metadata[\"url\"])\n",
        "    print(\"Score\\t\", src.score)\n",
        "    print(\"-_\" * 20)"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "authorship_tag": "ABX9TyOUem37lhhg0mJYauho+pvb",
      "include_colab_link": true,
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.12.4"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}