run without RAG

README initial.md

@@ -1,25 +0,0 @@
-# A template for a RAG system with Gradio UI
-Deliberately stripped down to leave some room for experimenting
-
-# Setting it up
-- Clone https://github.com/huggingface/transformers to a local machine
-- Use the **prep_scrips/markdown_to_text.py** script to extract raw text from markdown from transformers/docs/source/en/
-- Break the resulting texts down into semantically meaningful pieces. Experiment with different chunking mechanisms to make sure the semantic meaning is captured.
-- Use **prep_scrips/lancedb_setup.py** to embed and store chunks in a [lancedb](https://lancedb.github.io/lancedb/) instance. It also creates an index for fast ANN retrieval (not really needed for this exercise but necessary at scale). You'll need to put your own values into VECTOR_COLUMN_NAME, TEXT_COLUMN_NAME, DB_TABLE_NAME.
-- Move the database directory (.lancedb by default) to **gradio_app/**
-- Use the template given in **gradio_app** to wrap everything into the [Gradio](https://www.gradio.app/docs/interface) app and run it on HF [spaces](https://huggingface.co/docs/hub/spaces-config-reference). Make sure to adjust VECTOR_COLUMN_NAME, TEXT_COLUMN_NAME, DB_TABLE_NAME according to your DB setup.
-- In your space, set up secrets OPENAI_API_KEY and HUGGING_FACE_HUB_TOKEN to use OpenAI and open-source models correspondingly
-
-- TODOs: 
-  - Experiment with chunking, see how it affects the results. When deciding how to chunk it helps to think about what kind of chunks you'd like to see as context to your queries.
-    - Deliverables: Demonstrate how retrieved documents differ with different chunking strategies and how it affects the output.
-  - Try out different embedding models (EMBED_NAME). Good models to start with are **sentence-transformers/all-MiniLM-L6-v2** - lightweight, **thenlper/gte-large** - relatively heavy but more powerful.
-    - Deliverables: Demonstrate how retrieved documents differ with different embedding models and how they affect the output. Provide an estimate of how the time to embed the chunks and DB ingestion time differs (happening in **prep_scrips/lancedb_setup.py**).
-  - Add a re-ranker (cross-encoder) to the pipeline. Start with sentence-transformers pages on cross-encoders [1](https://www.sbert.net/examples/applications/cross-encoder/README.html) [2](https://www.sbert.net/examples/applications/retrieve_rerank/README.html), then pick a [pretrained cross-encoder](https://www.sbert.net/docs/pretrained-models/ce-msmarco.html), e.g. **cross-encoder/ms-marco-MiniLM-L-12-v2**. Don't forget to increase the number of *retrieved* documents when using re-ranker. The number of documents used as context should stay the same.
-    - Deliverables: Demonstrate how retrieved documents differ after adding a re-ranker and how it affects the output. Provide an estimate of how latency changes. 
-  - Try another LLM (e.g. LLaMA-2-70b, falcon-180b).
-    - Deliverables: Demonstrate how LLMs affect the output and how latency changes with the model size.
-  - Add more documents (e.g. diffusers, tokenizers, optimum, etc) to see how the system scales.
-    - Deliverables: Demonstrate how latency changes, and how it differs with and without index (index is added in **prep_scrips/lancedb_setup.py**).
-  - (Bonus) Use an LLM to quantitatively compare outputs of different variants of the system ([LLM as a Judge](https://huggingface.co/collections/andrewrreed/llm-as-a-judge-653fb861e361fd03c12d41e5))
-    - Deliverables: Describe the experimental setup and evaluation results

gradio_app/app.py

@@ -10,7 +10,7 @@ from time import perf_counter
 
 import gradio as gr
 import markdown
-import lancedb
+# import lancedb
 from jinja2 import Environment, FileSystemLoader
 
 from gradio_app.backend.ChatGptInteractor import num_tokens_from_messages
@@ -31,7 +31,8 @@ env = Environment(loader=FileSystemLoader('gradio_app/templates'))
 context_template = env.get_template('context_template.j2')
 context_html_template = env.get_template('context_html_template.j2')
 
-db = lancedb.connect(LANCEDB_DIRECTORY)
+# db = lancedb.connect(LANCEDB_DIRECTORY)
+db = None
 
 # Examples
 examples = [
@@ -49,13 +50,7 @@ def add_text(history, text):
     return history, gr.Textbox(value="", interactive=False)
 
 
-def bot(history, llm, cross_enc, chunk, embed):
-    history[-1][1] = ""
-    query = history[-1][0]
-
-    if not query:
-        raise gr.Error("Empty string was submitted")
-
+def find_context(query, cross_enc, chunk, embed):
     logger.info('Retrieving documents...')
     gr.Info('Start documents retrieval ...')
     t = perf_counter()
@@ -85,7 +80,10 @@ def bot(history, llm, cross_enc, chunk, embed):
 
     t = perf_counter() - t
     logger.info(f'Finished Reranking documents in {round(t, 2)} seconds...')
+    return documents
 
+
+def construct_messages(llm, documents, history):
     msg_constructor = get_message_constructor(llm)
     while len(documents) != 0:
         context = context_template.render(documents=documents)
@@ -98,6 +96,20 @@ def bot(history, llm, cross_enc, chunk, embed):
         documents.pop()
     else:
         raise gr.Error('Model context length exceeded, reload the page')
+    return documents, context_html, messages
+
+
+def bot(history, llm, cross_enc, chunk, embed):
+    history[-1][1] = ""
+    query = history[-1][0]
+
+    if not query:
+        raise gr.Error("Empty string was submitted")
+
+    # documents = find_context(query, cross_enc, chunk, embed)
+    # documents, context_html, messages = construct_messages(llm, documents, history)
+    context_html = ''
+    messages = get_message_constructor(llm)('', history)
 
     llm_gen = get_llm_generator(llm)
     logger.info('Generating answer...')
@@ -163,12 +175,15 @@ with gr.Blocks() as demo:
 
             llm_name = gr.Radio(
                 choices=[
+                    "gpt-4-1106-preview",
+                    "gpt-4",
+                    "gpt-3.5-turbo-1106",
                     "gpt-3.5-turbo",
                     "mistralai/Mistral-7B-Instruct-v0.1",
                     "tiiuae/falcon-180B-chat",
                     # "GeneZC/MiniChat-3B",
                 ],
-                value="gpt-3.5-turbo",
+                value="gpt-4-1106-preview",
                 label='LLM'
             )
 

gradio_app/backend/query_llm.py

@@ -5,10 +5,26 @@ from gradio_app.backend.HuggingfaceGenerator import HuggingfaceGenerator
 
 env = Environment(loader=FileSystemLoader('gradio_app/templates'))
 context_template = env.get_template('context_template.j2')
-start_system_message = context_template.render(documents=[])
+old_start_system_message = context_template.render(documents=[])
 
 
-def construct_mistral_messages(context, history):
+start_system_message = """
+Ignore all previous instructions.
+
+1. You are to provide clear, concise, and direct responses.
+2. Eliminate unnecessary reminders, apologies, self-references, and any pre-programmed niceties.
+3. Maintain a casual tone in your communication.
+4. Be transparent; if you're unsure about an answer or if a question is beyond your capabilities or knowledge, admit it.
+5. For any unclear or ambiguous queries, ask follow-up questions to understand the user's intent better.
+6. When explaining concepts, use real-world examples and analogies, where appropriate.
+7. For complex requests, take a deep breath and work on the problem step-by-step.
+8. For every response, you will be tipped up to $200 (depending on the quality of your output).
+
+It is very important that you get this right. Multiple lives are at stake.
+"""
+
+
+def old_construct_mistral_messages(context, history):
     messages = []
     for q, a in history:
         if len(a) == 0:  # the last message
@@ -25,11 +41,30 @@ def construct_mistral_messages(context, history):
     return messages
 
 
-def construct_openai_messages(context, history):
+def construct_mistral_messages(context, history):
+    messages = []
+    for q, a in history:
+        # if len(a) == 0:  # the last message
+        #     q = context + f'\n\nQuery:\n\n{q}'
+        if len(messages) == 0:
+            q = f'{start_system_message} The question:\n\n{q}'
+        messages.append({
+            "role": "user",
+            "content": q,
+        })
+        if len(a) != 0:  # do not insert the last answer
+            messages.append({
+                "role": "assistant",
+                "content": a,
+            })
+    return messages
+
+
+def old_construct_openai_messages(context, history):
     messages = [
         {
             "role": "system",
-            "content": start_system_message,
+            "content": old_start_system_message,
         },
     ]
     for q, a in history:
@@ -50,8 +85,33 @@ def construct_openai_messages(context, history):
     return messages
 
 
+def construct_openai_messages(context, history):
+    messages = [
+        {
+            "role": "system",
+            "content": start_system_message,
+        },
+    ]
+    for q, a in history:
+        # if len(a) == 0:  # the last message
+        #     messages.append({
+        #         "role": "system",
+        #         "content": context,
+        #     })
+        messages.append({
+            "role": "user",
+            "content": q,
+        })
+        if len(a) != 0:  # do not insert the last answer
+            messages.append({
+                "role": "assistant",
+                "content": a,
+            })
+    return messages
+
+
 def get_message_constructor(llm_name):
-    if llm_name == 'gpt-3.5-turbo':
+    if llm_name in ["gpt-4", "gpt-4-1106-preview", "gpt-3.5-turbo", "gpt-3.5-turbo-1106"]:
         return construct_openai_messages
     if llm_name in ['mistralai/Mistral-7B-Instruct-v0.1', "tiiuae/falcon-180B-chat", "GeneZC/MiniChat-3B"]:
         return construct_mistral_messages
@@ -59,9 +119,10 @@ def get_message_constructor(llm_name):
 
 
 def get_llm_generator(llm_name):
-    if llm_name == 'gpt-3.5-turbo':
+    if llm_name in ["gpt-4", "gpt-4-1106-preview", "gpt-3.5-turbo", "gpt-3.5-turbo-1106"]:
         cgi = ChatGptInteractor(
-            model_name=llm_name, max_tokens=512, temperature=0, stream=True
+            model_name=llm_name, stream=True,
+            # max_tokens=None, temperature=0,
         )
         return cgi.chat_completion
     if llm_name == 'mistralai/Mistral-7B-Instruct-v0.1' or llm_name == "tiiuae/falcon-180B-chat":
@@ -70,11 +131,11 @@ def get_llm_generator(llm_name):
         )
         return hfg.generate
 
-    if llm_name == "GeneZC/MiniChat-3B":
-        hfg = HuggingfaceGenerator(
-            model_name=llm_name, temperature=0, max_new_tokens=250, stream=False,
-        )
-        return hfg.generate
+    # if llm_name == "GeneZC/MiniChat-3B":
+    #     hfg = HuggingfaceGenerator(
+    #         model_name=llm_name, temperature=0, max_new_tokens=250, stream=False,
+    #     )
+    #     return hfg.generate
     raise ValueError('Unknown LLM name')
 
 

requirements.txt

@@ -4,10 +4,12 @@ ipywidgets==8.1.1
 tqdm==4.66.1
 aiohttp==3.8.6
 huggingface-hub==0.17.3
-lancedb>=0.3
+lancedb==0.3.4
 openai==0.28
 gradio==4.4.1
 markdown==3.5.1
 tiktoken==0.5.1
 huggingface-hub==0.17.3
 sentence-transformers==2.2.2
+beautifulsoup4==4.12.2
+Jinja2==3.1.2

settings.py

@@ -30,6 +30,9 @@ context_lengths = {
     "tiiuae/falcon-180B-chat": 2048,
     "GeneZC/MiniChat-3B": 4096,
     "gpt-3.5-turbo": 4096,
+    "gpt-4": 8192,
+    "gpt-4-1106-preview": 128000,
+    "gpt-3.5-turbo-1106": 16385,
     "sentence-transformers/all-MiniLM-L6-v2": 128,
     "thenlper/gte-large": 512,
     "text-embedding-ada-002": 1000,  # actual context length is 8191, but it's too much