readme

README.md

@@ -1,46 +1,47 @@
-# Meta Expert
+# Super Expert
 
 A project for versatile AI agents that can run with proprietary models or completely open-source. The meta expert has two agents: a basic [Meta Agent](Docs/Meta-Prompting%20Overview.MD), and [Jar3d](Docs/Introduction%20to%20Jar3d.MD), a more sophisticated and versatile agent.
 
-Act as an opne source perplexity.
+Act as an open source perplexity.
 
 Thanks John Adeojo, who brings this wonderful project to open source community!
 
-## PMF - What problem this project has solved?
+## Tech Stack
+- LLM(openai, claude, llama)
+- Frontend(Chainlit - chain of thought reasoning)
+- Backend
+   - python
+   - docker  
+   - Hugging Face deploy
+
+## TODO
+[] Long-term memory.
+[] Full Ollama and vLLM integration.
+[] Integrations to RAG platforms for more intelligent document processing and faster RAG.
 
-## Technical Detail
-What is the logics?
+## PMF - What problem this project has solved?
 
-LLM Application Workflow
+## Business Logics
+### LLM Application Workflow
 1. User Query: The user initiates the interaction by submitting a query or request for information.
 2. Agent Accesses the Internet: The agent retrieves relevant information from various online sources, such as web pages, articles, and databases.
-3. Document Chunking: The retrieved URLs are processed to break down the content into smaller, manageable documents or chunks. This step ensures that the information is more digestible and can be analyzed effectively.
+3. Document Chunking: The retrieved URLs are processed to break down the content into smaller, manageable documents or chunks. This step ensures that the information is more digestible and can be analyzed effectively.(tools\legacy\offline_graph_rag_tool copy.py run_rag)
 4. Vectorization: Each document chunk is then transformed into a multi-dimensional embedding using vectorization techniques. This process captures the semantic meaning of the text, allowing for nuanced comparisons between different pieces of information.
 5. Similarity Search: A similarity search is performed using cosine similarity (or another appropriate metric) to identify and rank the most relevant document chunks in relation to the original user query. This step helps in finding the closest matches based on the embeddings generated earlier.
 6. Response Generation: Finally, the most relevant chunks are selected, and the LLM synthesizes them into a coherent response that directly addresses the user's query.
 
 ## Bullet points
-- By implemented RAG, Chain-of-Reasoning, and Meta-Prompting to complete long-running research tasks.
-
-- Neo4j Knowledge Graphs
-   -Why use this?
-   naive RAG:
-   ![naive](image.png)
-   Complex:
-   ![why need graph](assets/image.png)
-
 
-- Docker for backend
-
-- NLM-Ingestor - llmsherpa API - Chunk data
 
+## FAQ
+1. How this system work?
 
+2. 
 
 
-## FAQ
-1. Is it necessary for a recursion more than 30 rounds? Is it spending money too much?
-
 
+2. How hybrid-retrieval work?
+In `offline_graph_rag` file, we combine similarity search with
 
 ## Table of Contents
 
@@ -115,6 +116,7 @@ This project leverages four core concepts:
    nano config/config.yaml
    ```
 
+If you want to use hyhbrid search, please open settings and choose "Graph and Dense".
 ### API Key Configuration
 
 Enter API Keys for your choice of LLM provider:
@@ -222,9 +224,3 @@ Refer to the project's GitHub issues for common problems and solutions.
 
 Once you're set up, Jar3d will proceed to introduce itself and ask some questions. The questions are designed to help you refine your requirements. When you feel you have provided all the relevant information to Jar3d, you can end the questioning part of the workflow by typing `/end`.
 
-## Roadmap for Jar3d
-
-- Feedback to Jar3d so that final responses can be iterated on and amended.
-- Long-term memory.
-- Full Ollama and vLLM integration.
-- Integrations to RAG platforms for more intelligent document processing and faster RAG.

tools/legacy/offline_graph_rag_tool copy.py

@@ -1,3 +1,5 @@
+# Hybird RAG, combining "similarity search" & "knowledge graph"
+
 import sys
 import os
 root_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
@@ -224,6 +226,7 @@ def run_hybrid_graph_retrrieval(graph: Neo4jGraph = None, corpus: List[Document]
         print(colored("Running Hybrid Retrieval...", "yellow"))
         unstructured_data = index_and_rank(corpus, query)
 
+        # We only feed > 30 to jar3d, subset
         query = f"""
         MATCH p = (n)-[r]->(m)
         WHERE COUNT {{(n)--()}} > 30
@@ -241,6 +244,7 @@ def run_hybrid_graph_retrrieval(graph: Neo4jGraph = None, corpus: List[Document]
     return retrieved_context
 
 
+# The chunking process begins with the intelligent_chunking function, which takes a URL and a query as input parameters.
 @timeout(20)  # Change: Takes url and query as input
 def intelligent_chunking(url: str, query: str) -> List[Document]:
     try:
@@ -251,7 +255,9 @@ def intelligent_chunking(url: str, query: str) -> List[Document]:
             raise ValueError("LLM_SHERPA_SERVER environment variable is not set")
         
         corpus = []
-
+#The function utilizes LayoutPDFReader to read and extract text from the specified PDF document located at the given URL. 
+#This is done by calling the LLM Sherpa API, which handles the PDF reading and layout analysis.
+#
         try: 
             print(colored("Starting LLM Sherpa LayoutPDFReader...\n\n", "yellow"))
             reader = LayoutPDFReader(llmsherpa_api_url)
@@ -261,7 +267,7 @@ def intelligent_chunking(url: str, query: str) -> List[Document]:
             print(colored(f"Error in LLM Sherpa LayoutPDFReader: {str(e)}", "red"))
             traceback.print_exc()
             doc = None
-        
+# Once the document is retrieved, it is processed into smaller, manageable chunks. Each chunk represents a segment of the document that retains semantic meaning and context.
         if doc:
             for chunk in doc.chunks():
                 document = Document(
@@ -321,6 +327,8 @@ def create_graph_index(
 ) -> Neo4jGraph:
     
     if os.environ.get('LLM_SERVER') == "openai":
+        # require hundreds calls to api
+        # we create index for every small chunk
         llm = ChatOpenAI(temperature=0, model_name="gpt-4o-mini")
 
     else:
@@ -370,6 +378,7 @@ def create_graph_index(
 
 def run_rag(urls: List[str], allowed_nodes: list[str] = None, allowed_relationships: list[str] = None, query: List[str] = None, hybrid: bool = False) -> List[Dict[str, str]]:
     # Change: adapted to take query and url as input.
+    # Intellegent document chunking
     with concurrent.futures.ThreadPoolExecutor(max_workers=min(len(urls), 5)) as executor:  
             futures = [executor.submit(intelligent_chunking, url, query) for url, query in zip(urls, query)]
             chunks_list = [future.result() for future in concurrent.futures.as_completed(futures)]
@@ -382,6 +391,7 @@ def run_rag(urls: List[str], allowed_nodes: list[str] = None, allowed_relationsh
 
     print(colored(f"\n\n DEBUG HYBRID VALUE: {hybrid}\n\n", "yellow"))
     
+    # combined with graph
     if hybrid:
         print(colored(f"\n\n Creating Graph Index...\n\n", "green"))
         graph = Neo4jGraph()

tools/offline_graph_rag_tool.py

@@ -67,7 +67,7 @@ def deduplicate_results(results, rerank=True):
             unique_results.append(result)
     return unique_results
 
-
+# Similarity search
 def index_and_rank(corpus: List[Document], query: str, top_percent: float = 20, batch_size: int = 25) -> List[Dict[str, str]]:
     print(colored(f"\n\nStarting indexing and ranking with FastEmbeddings and FAISS for {len(corpus)} documents\n\n", "green"))
     CACHE_DIR = "/app/fastembed_cache"
@@ -78,12 +78,13 @@ def index_and_rank(corpus: List[Document], query: str, top_percent: float = 20,
     try:
         # Initialize an empty FAISS index
         index = None
-        docstore = InMemoryDocstore({})
+        docstore = InMemoryDocstore({}) # store meta data
         index_to_docstore_id = {}
 
         # Process documents in batches
         for i in range(0, len(corpus), batch_size):
             batch = corpus[i:i+batch_size]
+            # abstract content and metadata
             texts = [doc.page_content for doc in batch]
             metadatas = [doc.metadata for doc in batch]
 
@@ -215,13 +216,18 @@ def index_and_rank(corpus: List[Document], query: str, top_percent: float = 20,
 
     return final_results
 
+# TODO optimize the retrieval
 def run_hybrid_graph_retrieval(graph: Neo4jGraph = None, corpus: List[Document] = None, query: str = None, hybrid: bool = False):
     print(colored(f"\n\Initiating Retrieval...\n\n", "green"))
 
     if hybrid:
         print(colored("Running Hybrid Retrieval...", "yellow"))
-        unstructured_data = index_and_rank(corpus, query)
+        # Similarity search
+        unstructured_data = index_and_rank(corpus, query) # similarity ranking 
 
+        # Cypher query language
+        # eg: where there is a directed relationship from node n to node m through relationship r.
+        # This condition filters the results to include only those nodes n that have more than 30 connections (relationships) to other nodes. The {(n)--()} syntax counts all relationships connected to node n, ensuring that only well-connected nodes are considered.
         query = f"""
         MATCH p = (n)-[r]->(m)
         WHERE COUNT {{(n)--()}} > 30
@@ -229,6 +235,7 @@ def run_hybrid_graph_retrieval(graph: Neo4jGraph = None, corpus: List[Document]
         LIMIT 85
         """
         response = graph.query(query)
+        # this context will then pass to LLM to generate response
         retrieved_context = f"Important Relationships:{response}\n\n Additional Context:{unstructured_data}"
 
     else: