Tested summary refine using ollama locally

examples/techniques/Contextual_RAG.ipynb

@@ -112,7 +112,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 26,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -121,7 +121,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 22,
+   "execution_count": 27,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -166,7 +166,16 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 23,
+   "execution_count": 28,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "output_text = result[output_key]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 29,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -175,31 +184,39 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 24,
+   "execution_count": 30,
    "metadata": {},
    "outputs": [
     {
      "data": {
       "text/markdown": [
-       "Here is a summary of the text:\n",
+       "Here is a comprehensive summary of the text:\n",
        "\n",
-       "A state machine is a mathematical model that describes how an output signal is generated from an input signal step-by-step. It consists of five main components: \n",
+       "A state machine is a mathematical model that represents an automaton or computer program that operates on inputs and produces outputs in steps. The 5-tuple StateMachine = (States, Inputs, Outputs, update, initialState) defines these five components.\n",
        "\n",
-       "1. States (representing different states or conditions)\n",
-       "2. Inputs (input signals, such as letters or symbols)\n",
-       "3. Outputs (output signals, which represent the actual output based on the input and state)\n",
-       "4. Update function (a way to modify the current state based on the inputs and outputs)\n",
-       "5. Initial State (the starting point of the machine)\n",
+       "**States**: Represent the current state of the system.\n",
+       "**Inputs**: Represent the input signals that affect the system's behavior.\n",
+       "**Outputs**: Represent the output signals produced by the system as a result of processing the inputs.\n",
+       "**update**: A function that determines the next state based on the current state and input.\n",
+       "**initialState**: The initial state of the system, which defines its starting point.\n",
        "\n",
-       "An example is given where a state machine is defined with three states: States, Inputs, Outputs. The initial state is also provided as an option.\n",
+       "A state machine can be thought of as a sequence of steps where each step represents one symbol in an infinite sequence of symbols (e.g., x(n)) being processed. This sequence defines the function that describes how to process those inputs to produce outputs.\n",
        "\n",
-       "The key points are:\n",
+       "The key characteristics of a state machine are:\n",
        "\n",
-       "* Time is not involved in this model; instead, step numbers refer to the order in which steps occur.\n",
-       "* Each input signal can be represented by an infinite sequence of symbols, such as a natural number sequence (e.g., 0 -> Inputs).\n",
-       "* The state machine evolves or \"moves\" from one state to another based on the inputs and outputs.\n",
+       "* One event occurs before another event\n",
+       "* Time is not specified between events\n",
+       "* The system can have different states at any given time\n",
        "\n",
-       "This model is used for various applications, including control systems, data processing, and communication systems."
+       "State machines can be used to model complex systems, such as communication protocols, computer algorithms, and control systems. They provide a mathematical framework for analyzing and designing these systems.\n",
+       "\n",
+       "The 5-tuple StateMachine = (States, Inputs, Outputs, update, initialState) is the most common way to represent a state machine in mathematical terms. It consists of:\n",
+       "\n",
+       "* States: The set of possible states\n",
+       "* Inputs: The set of possible input symbols\n",
+       "* Outputs: The set of possible output symbols\n",
+       "* update: A function that determines the next state based on the current state and input\n",
+       "* initialState: The initial state of the system"
       ],
       "text/plain": [
        "<IPython.core.display.Markdown object>"
@@ -210,15 +227,18 @@
     }
    ],
    "source": [
-    "display(Markdown(result[output_key]))"
+    "display(Markdown(output_text))"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 31,
    "metadata": {},
    "outputs": [],
-   "source": []
+   "source": [
+    "with open(\"data/document_summary.txt\", \"w\", encoding=\"utf-8\") as fp:\n",
+    "    fp.write(output_text)"
+   ]
   },
   {
    "cell_type": "markdown",
@@ -239,6 +259,28 @@
     "The best approach will be to use local models to achive this kind of heavy inference. For that we will turn to either **Ollama** or hugging face **Transformers**."
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "vscode": {
+     "languageId": "html"
+    }
+   },
+   "source": [
+    "<h2 align=center> Proper Contextual RAG </h2>\n",
+    "\n",
+    "NOw that we have an executive summary to provide the context for each chunk, we proceed to the main implementation."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 32,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "document_summary = output_text"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,

examples/techniques/Contextual_RAG.py

@@ -0,0 +1,112 @@
+# %% [markdown]
+# <h1 align=center> Contextual RAG </h1>
+# 
+# ![anthropic blog poas](https://www.anthropic.com/_next/image?url=https%3A%2F%2Fwww-cdn.anthropic.com%2Fimages%2F4zrzovbb%2Fwebsite%2F2496e7c6fedd7ffaa043895c23a4089638b0c21b-3840x2160.png&w=3840&q=75)
+# 
+# This is an approach proposed by Anthropic in a recent [blog poas](https://www.anthropic.com/news/contextual-retrieval). It involves improving retrieval by providing each document chunk with an in context summary.
+
+# %% [markdown]
+# <h2 align=center> Problems </h2>
+# 
+# As one may gather from the explanation, there is a requirement that each chunk be appropriately contextualized with respect to the rest of the document. So essentially the whole document has to be passed into the prompt each time along with the chunk. There are two problems with this:
+# 
+# 1. This would be very expensive in terms of input token count.
+# 2. For models with smaller context windows, the whole document may exceed it.( Further, there is a sense in which fitting a whole document into a models context width defeats the point of performing RAG.)
+# 
+
+# %% [markdown]
+# <h2 align=center> Whole Document Summarization </h2>
+# 
+# The solution I have come up with is to instead summarize the document into a more manageable size.
+
+# %% [markdown]
+# <h3 align=center> Refine </h3>
+
+# %%
+from langchain.chains.combine_documents.stuff import StuffDocumentsChain
+from langchain.chains.llm import LLMChain
+from langchain.prompts import PromptTemplate
+from langchain_text_splitters import CharacterTextSplitter
+from langchain.document_loaders import PyMuPDFLoader
+
+# %%
+from langchain.chains.summarize import load_summarize_chain
+
+# %%
+# from langchain_google_genai import ChatGoogleGenerativeAI
+# import os
+# from dotenv import load_dotenv
+
+# if not load_dotenv():
+#     print("API keys may not have been loaded succesfully")
+# google_api_key = os.getenv("GOOGLE_API_KEY")
+
+# llm = ChatGoogleGenerativeAI(model="gemini-pro", api_key=google_api_key)
+
+# %%
+from langchain_ollama.llms import OllamaLLM
+
+# A lightweigh model for local inference
+llm = OllamaLLM(model="llama3.2:1b-instruct-q4_K_M")
+
+# %%
+loader = PyMuPDFLoader("data/State Machines.pdf")
+docs = loader.load()
+
+# %%
+text_splitter = CharacterTextSplitter.from_tiktoken_encoder(chunk_size=1000, chunk_overlap=0)
+split_docs = text_splitter.split_documents(docs)
+
+# %%
+prompt = """
+                  Please provide a summary of the following text.
+                  TEXT: {text}
+                  SUMMARY:
+                  """
+
+question_prompt = PromptTemplate(
+    template=prompt, input_variables=["text"]
+)
+
+refine_prompt_template = """
+              Write a concise summary of the following text delimited by triple backquotes.
+              Return your response in bullet points which covers the key points of the text.
+              ```{text}```
+              BULLET POINT SUMMARY:
+              """
+
+refine_template = PromptTemplate(
+    template=refine_prompt_template, input_variables=["text"]
+)
+
+# Load refine chain
+chain = load_summarize_chain(
+    llm=llm,
+    chain_type="refine",
+    question_prompt=question_prompt,
+    refine_prompt=refine_template,
+    return_intermediate_steps=True,
+    input_key="input_documents",
+    output_key="output_text",
+)
+result = chain({"input_documents": split_docs}, return_only_outputs=True)
+
+# %% [markdown]
+# <h3 align=center> Remarks </h3>
+# 
+# Refine is properly configured but we ran into this error.
+# 
+# ```python
+# ResourceExhausted: 429 Resource has been exhausted (e.g. check quota).
+# ```
+# 
+# This is a problem on the part of our llm provider not the code.
+# 
+# <h3 align=center> Next Steps </h3>
+# 
+# The best approach will be to use local models to achive this kind of heavy inference. For that we will turn to either **Ollama** or hugging face **Transformers**.
+
+# %%
+
+
+

modules/ollama_sum_refine.py

@@ -0,0 +1,105 @@
+from langchain.prompts import PromptTemplate
+from langchain_text_splitters import RecursiveCharacterTextSplitter
+from langchain.chains.summarize import load_summarize_chain
+from langchain.docstore.document import Document
+
+from langchain_ollama.llms import OllamaLLM
+
+
+llm = OllamaLLM(model="llama3.2:1b-instruct-q4_K_M")
+
+def refine(text: str, prompt=None, refine_prompt=None) -> str:
+    """
+        use the refine method to summarize text. More can be learned here: 
+            https://python.langchain.com/v0.1/docs/use_cases/summarization/#option-3-refine
+    """
+    text_splitter = RecursiveCharacterTextSplitter(separators=["\n\n\n", "\n\n", "\n"], 
+                                                #    chunk_size=128, chunk_overlap=0
+                                                   )
+    split_texts = text_splitter.split_text(text)
+
+    # Convert into langchain docs for downstream chain
+    split_docs =  []
+    for text in split_texts:
+        page = Document(page_content=text,
+                        metadata = {"source": "local"})
+        split_docs.append(page)
+
+
+    if prompt is None:
+        prompt = """
+                        Please provide a very comprehensive summary of the following text.
+                        WHile maintaining lower level detail
+                        
+                        TEXT: {text}
+
+                        Begin by summarizing the topic at hand briefly
+                        in the same way an abstract explains a paper
+
+                        SUMMARY:
+                        """
+
+    question_prompt = PromptTemplate(
+        template=prompt, input_variables=["text"]
+    )
+
+    if refine_prompt is None:
+        refine_prompt = """
+            You are tasked with refining and improving an existing summary. We have an initial summary that is accurate but may lack details from the new context below.
+
+            ---
+            Existing Summary:
+            {existing_answer}
+
+            New Context:
+            {text}
+            ---
+
+            Please refine the existing summary by incorporating relevant information from the new context. Ensure the refined summary remains clear, concise, and cohesive. If the new context does not provide useful details, keep the original summary unchanged. Avoid repeating information unnecessarily. Return the improved summary below.
+            s
+            """
+
+
+    refine_template = PromptTemplate(
+        template=refine_prompt, input_variables=["text", "existing_answer"]
+    )
+
+    output_key = "output_text"
+    
+    # Load refine chain
+    chain = load_summarize_chain(
+        llm=llm,
+        chain_type="refine",
+        question_prompt=question_prompt,
+        refine_prompt=refine_template,
+        return_intermediate_steps=True,
+        input_key="input_documents",
+        output_key=output_key,
+    )
+    result = chain({"input_documents": split_docs}, return_only_outputs=True)
+    return result[output_key]
+
+if __name__ == "__main__":
+    long_text = """
+        And Hector quickly reached for his son. But the boy
+        recoiled, crying out to his nurse,
+        terrified by his father’s bronze-encased appearance—
+        the crest of the horsehair helmet
+        shone so bright it frightened him.
+        At that, Hector and his wife both burst out laughing,
+        and from his head Hector lifted off the helmet,
+        and set it on the ground, all shimmering with light.
+        Then he kissed his dear son, tossing him in his arms,
+        lifting a prayer to Zeus and the other gods:
+        'Zeus, and all gods, grant this boy of mine
+        to be, like me, preeminent in Troy,
+        strong and brave, and ruling Ilium with might.
+        Then one day men will say of him,
+        as he returns from war, bearing the bloodstained gear of slaughtered foes,
+        "A far better man than his father!"'
+        And Hector placed his son in his wife's arms,
+        and she embraced him, smiling through her tears.
+    """
+
+    print(refine(long_text))
+

ollama_inference.py

@@ -0,0 +1,14 @@
+import streamlit as st
+from modules.ollama_sum_refine import refine
+
+st.write("### Get an Ollama Response ✨")
+
+text = st.text_area("Input your Text tHere", height=300)
+
+@st.cache_data
+def summarize(content):
+    return refine(content)
+
+if text:
+    response = summarize(text)
+    response