Adds all necessary files

.env

@@ -0,0 +1,2 @@
+TESSERACT_PATH=C:\Program Files\Tesseract-OCR\tesseract.exe
+CHROMA_PATH=./../chroma

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+chroma/chroma.sqlite3 filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -0,0 +1,12 @@
+.DS_Store
+.vscode
+venv/
+evaluationResults/*
+
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# Jupyter Notebook
+.ipynb_checkpoints

Dockerfile

@@ -0,0 +1,52 @@
+# The python builder image, used to build the virtual environment
+FROM python:3.11-slim-buster
+
+# Install system dependencies
+RUN apt-get update && apt-get install -y --no-install-recommends git && \
+    rm -rf /var/lib/apt/lists/*
+
+# Create a user to run the app
+RUN useradd -m -u 1000 user
+
+# Switch to user and set environment variables
+USER user
+ENV HOME=/home/user \
+    PATH="/home/user/.local/bin:$PATH" \
+    VIRTUAL_ENV=/home/user/venv \
+    LISTEN_PORT=7860 \
+    HOST=0.0.0.0
+
+# Set the working directory in the container
+WORKDIR $HOME
+
+# Create a virtual environment to isolate our package dependencies locally
+RUN python -m venv $VIRTUAL_ENV
+ENV PATH="$VIRTUAL_ENV/bin:$PATH"
+
+# Copy necessary files to container directory
+COPY --chown=user ./app ./app/app
+COPY --chown=user ./chroma ./chroma
+COPY --chown=user ./embedding_model ./embedding_model
+COPY --chown=user ./sparse_index ./sparse_index
+COPY --chown=user ./.env ./app/.env
+COPY --chown=user ./app/chainlit.md ./app/chainlit.md
+COPY --chown=user ./app/.chainlit ./app/.chainlit
+COPY --chown=user ./app/public ./app/public
+COPY --chown=user ./input_data/Templates/template_files ./input_data/Templates/template_files
+COPY --chown=user ./requirements_Docker.txt ./app/requirements_Docker.txt
+COPY --chown=user ./init_embedding_model.py ./init_embedding_model.py
+
+WORKDIR $HOME/app
+
+# Install Python dependencies
+RUN pip install --upgrade pip && \
+    pip install -r ./requirements_Docker.txt
+
+# Run the script to initialize and cache the fine-tuned embedding model
+RUN python ./init_model.py
+
+# Expose the port the app runs on
+EXPOSE $LISTEN_PORT
+
+# Run the chainlit app
+CMD ["chainlit", "run", "app/app.py", "--host", "0.0.0.0", "--port", "7860"]

README_PROJECT.md

@@ -0,0 +1,73 @@
+# General Information
+
+## 1. Project Initilization
+- After pulling the project, do the following to initialize the project:
+    - Make sure that a Python Version >= 3.11 is installed
+    - Run the following command to execute the initialization script: "source setup.sh"
+- If you want to insert new PDF documents and update the document base, you first need to install Tesseract which is the OCR engine used in this code:
+    - Download Tesseract Installer for Windows: https://github.com/UB-Mannheim/tesseract/wiki
+    - For others, see here: https://tesseract-ocr.github.io/tessdoc/Installation.html
+- Create a .env file on the root directory level, with the following keys:
+    - TESSERACT_PATH={path}
+        - Set path to the installation path of tesseract, e.g. "C:\Program Files\Tesseract-OCR\tesseract.exe"
+    - CHROMA_PATH=./../chroma
+
+## 2. Using the Chatbot locally
+- In the app/helper.py file comment out lines 8 to 10 if you are not on a Linux machine
+- To start the chatbot locally, run "cd app" and "chainlit run app.py -w"
+- To use the chatbot, you need two API keys which you can create under the following links
+    - [OpenAI](https://openai.com/blog/openai-api)
+    - [Cohere](https://dashboard.cohere.com/api-keys)
+
+## 3. Using Docker
+- Go to helper.py and uncomment the three lines for the import. This is necessary to use Chroma within the container
+- Build the docker file: "docker build -t iso_27001_chatbot ."
+- Running the docker file: "docker run -p 7860:7860 iso_27001_chatbot"
+    - Access at: http://localhost:7860
+- Note that the docker file uses the requirements_Docker.txt which do not include Cuda support, as the free version of HF spaces does not come with GPU availability. If you want to include Cuda support, you need to integrate the command seen above for installing torch into the Dockerfile.
+
+# Project Structure
+
+## app
+Contains the chatbot web application, created with Chainlit. Also, includes classes for prompts and helper functions.
+
+## chroma
+The chroma folder consists of all the indices that were created by using the notebooks inside the index_preparation folder.
+
+## embedding_model
+This folder contains the embedding model fine-tuned on an ISO 27001 text corpus under. It is based on [bge-large-en-v.1.5](https://huggingface.co/BAAI/bge-large-en-v1.5) and can be accessed and downloaded on [HuggingFace](https://huggingface.co/Basti8499/bge-large-en-v1.5-ISO-27001).
+
+## index_preparation
+Stores all Jupyter notebooks needed to create the vector database which stores the ISO 27001 documents. Before creating the index with the build_index.ipynb, the documents for PDFs, Web and Templates need be created inside the other notebooks.
+
+## input_data
+
+### PDF Files (/PDF)
+- Directory structure:
+    - PDF/files: After manually cleaning the PDFs (removing pages), the PDFs should be moved manually to this folder.
+    - PDF/documents
+        - /all_documents: JSON file for all processed PDF documents
+        - /new_documents: JSON file for newly processed PDF documents
+    - PDF/PDF_images: Empty folder in which the images during OCR are stored and deleted afterwards.
+
+### Web Files (/Web)
+- Directory structure:
+    - Web/documents:
+        - /all_documents: JSON file for all processed web documents
+        - /newl_documents: JSON file for newly processed web documents
+    - Web/URLs:
+        - /cleaned_urls.txt: .txt file for URLs that were already processed and documents exist
+        - /uncleaned_urls.txt: .txt file for URLs that have not been processed
+
+### Template Files (/Templates)
+- Directory structure:
+    - Templates/documents:
+        - /all_documents: JSON file for all processed template documents
+        - /new_documents:  JSON file for all newly processed template documents
+    - Templates/template_files:
+        - /new: Not yet processed template files
+        - /processed: Already processed template files
+-  For templates it is important that the actual files to the template are stored under Templates/template_files/new for processing, as the paths are used in the chatbot.
+
+## sparse_index
+Stores the chunked documents that were created in the build_index.ipynb in a .txt file for later sparse retrieval.

app/.chainlit/config.toml

@@ -0,0 +1,97 @@
+[project]
+# Whether to enable telemetry (default: true). No personal data is collected.
+enable_telemetry = true
+
+
+# List of environment variables to be provided by each user to use the app.
+user_env = ["OPENAI_API_KEY","COHERE_API_KEY"]
+
+# Duration (in seconds) during which the session is saved when the connection is lost
+session_timeout = 3600
+
+# Enable third parties caching (e.g LangChain cache)
+cache = false
+
+# Authorized origins 
+allow_origins = ["*"]
+
+# Follow symlink for asset mount (see https://github.com/Chainlit/chainlit/issues/317)
+# follow_symlink = false
+
+[features]
+# Show the prompt playground
+prompt_playground = false
+
+# Process and display HTML in messages. This can be a security risk (see https://stackoverflow.com/questions/19603097/why-is-it-dangerous-to-render-user-generated-html-or-javascript)
+unsafe_allow_html = false
+
+# Process and display mathematical expressions. This can clash with "$" characters in messages.
+latex = false
+
+# Authorize users to upload files with messages
+multi_modal = false
+
+# Allows user to use speech to text
+[features.speech_to_text]
+    enabled = false
+    # See all languages here https://github.com/JamesBrill/react-speech-recognition/blob/HEAD/docs/API.md#language-string
+    # language = "en-US"
+
+[UI]
+# Name of the app and chatbot.
+name = "ISO 27001 Chatbot"
+
+# Show the readme while the thread is empty.
+show_readme_as_default = true
+
+# Description of the app and chatbot. This is used for HTML tags.
+# description = ""
+
+# Large size content are by default collapsed for a cleaner ui
+default_collapse_content = true
+
+# The default value for the expand messages settings.
+default_expand_messages = false
+
+# Hide the chain of thought details from the user in the UI.
+hide_cot = true
+
+# Link to your github repo. This will add a github button in the UI's header.
+# github = ".."
+
+# Specify a CSS file that can be used to customize the user interface.
+# The CSS file can be served from the public directory or via an external link.
+# custom_css = "/public/test.css"
+
+# Specify a Javascript file that can be used to customize the user interface.
+# The Javascript file can be served from the public directory.
+# custom_js = "/public/test.js"
+
+# Specify a custom font url.
+# custom_font = "https://fonts.googleapis.com/css2?family=Inter:wght@400;500;700&display=swap"
+
+# Override default MUI light theme. (Check theme.ts)
+[UI.theme]
+    #font_family = "Inter, sans-serif"
+[UI.theme.light]
+    #background = "#FAFAFA"
+    #paper = "#FFFFFF"
+
+    [UI.theme.light.primary]
+        #main = "#F80061"
+        #dark = "#980039"
+        #light = "#FFE7EB"
+
+# Override default MUI dark theme. (Check theme.ts)
+[UI.theme.dark]
+    #background = "#FAFAFA"
+    #paper = "#FFFFFF"
+
+    [UI.theme.dark.primary]
+        #main = "#F80061"
+        #dark = "#980039"
+        #light = "#FFE7EB"
+
+
+[meta]
+generated_by = "1.0.401"

app/.chainlit/translations/en-US.json

@@ -0,0 +1,155 @@
+{
+    "components": {
+        "atoms": {
+            "buttons": {
+                "userButton": {
+                    "menu": {
+                        "settings": "Settings",
+                        "settingsKey": "S",
+                        "APIKeys": "API Keys",
+                        "logout": "Logout"
+                    }
+                }
+            }
+        },
+        "molecules": {
+            "newChatButton": {
+                "newChat": "New Chat"
+            },
+            "tasklist": {
+                "TaskList": {
+                    "title": "\ud83d\uddd2\ufe0f Task List",
+                    "loading": "Loading...",
+                    "error": "An error occured"
+                }
+            },
+            "attachments": {
+                "cancelUpload": "Cancel upload",
+                "removeAttachment": "Remove attachment"
+            },
+            "newChatDialog": {
+                "createNewChat": "Create new chat?",
+                "clearChat": "This will clear the current messages and start a new chat.",
+                "cancel": "Cancel",
+                "confirm": "Confirm"
+            },
+            "settingsModal": {
+                "expandMessages": "Expand Messages",
+                "hideChainOfThought": "Hide Chain of Thought",
+                "darkMode": "Dark Mode"
+            }
+        },
+        "organisms": {
+            "chat": {
+                "history": {
+                    "index": {
+                        "lastInputs": "Last Inputs",
+                        "noInputs": "Such empty...",
+                        "loading": "Loading..."
+                    }
+                },
+                "inputBox": {
+                    "input": {
+                        "placeholder": "Type your message here..."
+                    },
+                    "speechButton": {
+                        "start": "Start recording",
+                        "stop": "Stop recording"
+                    },
+                    "SubmitButton": {
+                        "sendMessage": "Send message",
+                        "stopTask": "Stop Task"
+                    },
+                    "UploadButton": {
+                        "attachFiles": "Attach files"
+                    },
+                    "waterMark": {
+                        "text": "Built with"
+                    }
+                },
+                "Messages": {
+                    "index": {
+                        "running": "Running",
+                        "executedSuccessfully": "executed successfully",
+                        "failed": "failed",
+                        "feedbackUpdated": "Feedback updated",
+                        "updating": "Updating"
+                    }
+                },
+                "dropScreen": {
+                    "dropYourFilesHere": "Drop your files here"
+                },
+                "index": {
+                    "failedToUpload": "Failed to upload",
+                    "cancelledUploadOf": "Cancelled upload of",
+                    "couldNotReachServer": "Could not reach the server",
+                    "continuingChat": "Continuing previous chat"
+                },
+                "settings": {
+                    "settingsPanel": "Settings panel",
+                    "reset": "Reset",
+                    "cancel": "Cancel",
+                    "confirm": "Confirm"
+                }
+            },
+            "threadHistory": {
+                "sidebar": {
+                    "filters": {
+                        "FeedbackSelect": {
+                            "feedbackAll": "Feedback: All",
+                            "feedbackPositive": "Feedback: Positive",
+                            "feedbackNegative": "Feedback: Negative"
+                        },
+                        "SearchBar": {
+                            "search": "Search"
+                        }
+                    },
+                    "DeleteThreadButton": {
+                        "confirmMessage": "This will delete the thread as well as it's messages and elements.",
+                        "cancel": "Cancel",
+                        "confirm": "Confirm",
+                        "deletingChat": "Deleting chat",
+                        "chatDeleted": "Chat deleted"
+                    },
+                    "index": {
+                        "pastChats": "Past Chats"
+                    },
+                    "ThreadList": {
+                        "empty": "Empty..."
+                    },
+                    "TriggerButton": {
+                        "closeSidebar": "Close sidebar",
+                        "openSidebar": "Open sidebar"
+                    }
+                },
+                "Thread": {
+                    "backToChat": "Go back to chat",
+                    "chatCreatedOn": "This chat was created on"
+                }
+            },
+            "header": {
+                "chat": "Chat",
+                "readme": "Readme"
+            }
+        }
+    },
+    "hooks": {
+        "useLLMProviders": {
+            "failedToFetchProviders": "Failed to fetch providers:"
+        }
+    },
+    "pages": {
+        "Design": {},
+        "Env": {
+            "savedSuccessfully": "Saved successfully",
+            "requiredApiKeys": "Required API Keys",
+            "requiredApiKeysInfo": "To use this app, the following API keys are required. The keys are stored on your device's local storage."
+        },
+        "Page": {
+            "notPartOfProject": "You are not part of this project."
+        },
+        "ResumeButton": {
+            "resumeChat": "Resume Chat"
+        }
+    }
+}

app/app.py

@@ -0,0 +1,99 @@
+import chainlit as cl
+from helper import HelperMethods
+from pydantic.v1.error_wrappers import ValidationError
+from cohere.error import CohereAPIError
+
+COLLECTION_NAME = "ISO_27001_Collection"
+
+
+@cl.on_chat_start
+async def on_chat_start():
+    """
+    Is called when a new chat session is created. Adds an initial message and sets important objects into session state.
+    """
+
+    await cl.sleep(1)
+
+    msg = cl.Message(author="ISO 27001 - Assistant", content="Hello, do you have questions on ISO 27001? Feel free to ask me.")
+    await msg.send()
+
+    helper = HelperMethods()
+
+    try:
+        llm, MAX_CONTEXT_SIZE = await helper.get_LLM()
+    except ValidationError as e:
+        error_message = cl.ErrorMessage(
+            author="ISO 27001 - Assistant",
+            content="A validation error occurred. Please ensure the Open API_KEY is correctly set. You can navigate to the profile icon and then reset the keys. After that reload the page and try to ask the question again.",
+        )
+        await error_message.send()
+        return
+
+    state = {"llm": llm, "max_context_size": MAX_CONTEXT_SIZE, "vectordb": helper.get_index_vector_db(COLLECTION_NAME)}
+    cl.user_session.set("state_ISO", state)
+
+
+@cl.on_message
+async def on_message(message: cl.Message):
+    """
+    Is called when a new message is sent by the user. Executes the RAG pipeline (check english, retrieve contexts, check relevancy, check context size, prompt LLM)
+    """
+
+    state = cl.user_session.get("state_ISO")
+    helper = HelperMethods()
+    query = message.content
+
+    if helper.check_if_english(query):
+
+        try:
+            docs = helper.retrieve_contexts(state["vectordb"], query)
+        except CohereAPIError as e:
+            error_message = cl.ErrorMessage(
+                author="ISO 27001 - Assistant",
+                content="A validation error occurred. Please ensure the Cohere API_KEY is correctly set. You can navigate to the profile icon and then reset the keys. After that reload the page and try to ask the question again.",
+            )
+            await error_message.send()
+            return
+        
+        if helper.check_if_relevant(docs):
+            if helper.is_context_size_valid(docs, query, state["max_context_size"]):
+
+                msg = cl.Message(author="ISO 27001 - Assistant", content="")
+                await msg.send()
+
+                full_prompt, sources, template_path, template_source= helper.get_full_prompt_sources_and_template(docs, state["llm"], query)
+
+                try:
+                    stream = state["llm"].astream(full_prompt)
+                except ValidationError as e:
+                    error_message = cl.ErrorMessage(
+                        author="ISO 27001 - Assistant",
+                        content="A validation error occurred. Please ensure the Open API_KEY is correctly set. You can navigate to the profile icon and then reset the keys. After that reload the page and try to ask the question again.",
+                    )
+                    await error_message.send()
+                    return  
+
+                async for part in stream:
+                    await msg.stream_token(part.content)
+
+                if template_path == "":
+                    sources_str = "\n\nSources: \n" + sources
+                    msg.content += sources_str
+                    await msg.update()
+                else:
+                    sources_str = "\n\nSources: \n" + sources
+                    elements = [cl.File(name=template_source, path=template_path, display="inline")]
+                    msg.content += sources_str
+                    msg.elements = elements
+                    await msg.update()
+            else:
+                await cl.Message(
+                    author="ISO 27001 - Assistant",
+                    content="I am sorry. I cannot process your question, as it would exceed my token limit. Please try to reformulate your question, or ask something else.",
+                ).send()
+        else:
+            await cl.Message(author="ISO 27001 - Assistant", content="I am sorry. I cannot process your question, as it is not related to ISO 27001.").send()
+    else:
+        await cl.Message(
+            author="ISO 27001 - Assistant", content="I am sorry. I cannot process your question, as I can only answer questions written in English."
+        ).send()

app/chainlit.md

@@ -0,0 +1,7 @@
+# Welcome to the ISO 27001 Chatbot! 🤖
+
+Hello, this Chainlit application lets you chat with a Retrieval Augmented Generation Pipeline with the focus on ISO 27001. It will only answer English questions related to this topic. Please be aware that this is not a production ready system and do not fully trust the answers, as it can still include errors.
+
+The RAG Pipeline needs to load the underlying models, so you can start as soon as the Assistant greets you. However, to use the chatbot, you need two API keys which you can create under the following links: [OpenAI](https://openai.com/blog/openai-api) (for the generation) and [Cohere](https://dashboard.cohere.com/api-keys) (for the retrieval re-ranking). After creating both keys, you need to assign them. This can be done by clicking on the user profile and setting the keys. After that please reload the page.
+
+Please be advised that because this HF Space runs on the free tier, no GPU is available. That's why answering a question takes up to 40 seconds. With GPU support this would be reduced to below 10 seconds.

app/helper.py

@@ -0,0 +1,217 @@
+from langchain_community.embeddings import HuggingFaceEmbeddings
+from langchain_openai import ChatOpenAI
+from langchain.retrievers.document_compressors import CohereRerank
+from langchain_community.retrievers import BM25Retriever
+import tiktoken
+
+# ONLY USE WITH DOCKER, then uncomment
+import pysqlite3
+import sys
+sys.modules["sqlite3"] = sys.modules.pop("pysqlite3")
+
+import chromadb
+import chainlit as cl
+from langdetect import detect
+from langchain_community.vectorstores import Chroma
+from typing import List
+from typing import Tuple
+import os
+import json
+from langchain.docstore.document import Document
+from prompts import get_system_prompt, get_human_prompt, get_system_prompt_template, get_full_prompt
+
+class HelperMethods:
+    """
+    Helper class with all important methods for the RAG pipeline.
+    """
+
+    def __init__(self):
+        pass
+
+    def _get_embedding_model(self):
+        """
+        Gets the finetuned embedding model based on bge-large-en-v1.5
+        """
+        path = "Basti8499/bge-large-en-v1.5-ISO-27001"
+        model = HuggingFaceEmbeddings(model_name=path)
+        return model
+
+
+    async def get_LLM(self):
+        """
+        Initializes the gpt-4.5 16k LLM
+        """
+        llm = ChatOpenAI(model_name="gpt-3.5-turbo-0125", temperature=0, max_tokens=680, streaming=True, api_key=cl.user_session.get("env")["OPENAI_API_KEY"])
+        max_context_size = 16385
+        return llm, max_context_size
+
+
+    def get_index_vector_db(self, collection_name: str):
+        """
+        Gets the index vector base based on the collection name, if existent.
+        """
+        new_client = chromadb.PersistentClient(path=os.environ.get("CHROMA_PATH"))
+
+        # Check if collection already exists
+        collection_exists = True
+        try:
+            new_client.get_collection(collection_name)
+        except ValueError as e:
+            collection_exists = False
+
+        if not collection_exists:
+            raise Exception("Error, raised exception: Collection does not exist.")
+        else:
+            embedding_model = self._get_embedding_model()
+            vectordb = Chroma(client=new_client, collection_name=collection_name, embedding_function=embedding_model)
+
+            return vectordb
+
+
+    def _load_documents(self, file_path: str) -> List[Document]:
+        documents = []
+        with open(file_path, "r") as jsonl_file:
+            for line in jsonl_file:
+                data = json.loads(line)
+                obj = Document(**data)
+                documents.append(obj)
+        return documents
+
+    def check_if_english(self, query: str) -> bool:
+        """
+        Uses the langdetect library based on  Google's language-detection library to check which language the query is in.
+        Returns True if it is English.
+        """
+        language = detect(query)
+        return language == "en"
+        
+    def check_if_relevant(self, docs: List[Document]) -> bool:
+
+        relevance_scores = [doc.metadata["relevance_score"] for doc in docs]
+        avg_score = sum(relevance_scores) / len(relevance_scores)
+        return avg_score > 0.75
+
+    def retrieve_contexts(self, vectordb, query: str, k: int = 8, rerank_k: int = 50, dense_percent: float = 0.5) -> List[Document]:
+        """
+        Retrieves the documents from the vector database by using a hybrid approach (dense (similarity search) + sparse (BM25)) and the Cohere re-ranking endpoint.
+        """
+        dense_k = int(rerank_k * dense_percent)
+        sparse_k = rerank_k - dense_k
+
+        # Sparse Retrieval
+        sparse_documents = self._load_documents(f"./../sparse_index/sparse_1536_264")
+        bm25_retriever = BM25Retriever.from_documents(sparse_documents)
+        bm25_retriever.k = sparse_k
+        result_documents_BM25 = bm25_retriever.get_relevant_documents(query)
+
+        # Dense Retrieval
+        result_documents_Dense = vectordb.similarity_search(query, k=dense_k)
+
+        result_documents_all = []
+        result_documents_all.extend(result_documents_BM25)
+        result_documents_all.extend(result_documents_Dense)
+
+        # Only get unique documents and remove duplicates that were retrieved in both sparse and dense
+        unique_documents_dict = {}
+        for doc in result_documents_all:
+            if doc.page_content not in unique_documents_dict:
+                unique_documents_dict[doc.page_content] = doc
+        result_documents_unique = list(unique_documents_dict.values())
+
+        # Re-ranking with Cohere
+        compressor = CohereRerank(top_n=k, user_agent="langchain", cohere_api_key=cl.user_session.get("env")["COHERE_API_KEY"])
+        result_documents = compressor.compress_documents(documents=result_documents_unique, query=query)
+
+        return result_documents
+
+
+    def is_context_size_valid(self, contexts: List[Document], query: str, max_context_size: int) -> bool:
+        """
+        Checks if the context size is valid with the cl100k tokenizer which is used for OpenAI LLM's.
+        """
+        # Transform List[Document] into List[str]
+        concatenated_contexts = ""
+        for index, document in enumerate(contexts):
+            original_text = document.metadata.get("original_text", "")
+            # Replace curly brackets, as otherwise problems can be encountered with formatting the prompt
+            original_text = original_text.replace("{", "").replace("}", "")
+            concatenated_contexts += f"{index+1}. {original_text}\n\n"
+
+        if not query.endswith("?"):
+            query = query + "?"
+
+        # Get the prompts
+        system_str, system_prompt = get_system_prompt()
+        human_str, human_prompt = get_human_prompt(concatenated_contexts, query)
+        full_prompt = system_str + "\n" + human_str
+
+        # Count token length
+        tokenizer = tiktoken.get_encoding("cl100k_base")
+        token_length = len(tokenizer.encode(full_prompt))
+
+        if token_length <= max_context_size:
+            return True
+        else:
+            return False
+
+    def get_full_prompt_sources_and_template(self, contexts: List[Document], llm, prompt: str) -> Tuple[str, str, str, str]:
+
+        # Check if the query is aimed at a template and check if the context documents also have a template
+        # If it is a template question the query and system prompt has to be altered
+        # Only check first two because otherwise the re-ranked score is not high enough to assume that the retrieved template is valid for that question
+        is_template_question = False
+        template_path = ""
+        template_source = ""
+        if "template" in prompt.lower():
+            for context in contexts[:2]:
+                if "template_path" in context.metadata:
+                    is_template_question = True
+                    template_path = context.metadata["template_path"]
+                    template_source = context.metadata["source"]
+                    break
+
+        # Concatenate all document texts and sources
+        concatenated_contexts = ""
+        concatenated_sources = ""
+        seen_sources = set()       
+        if is_template_question:
+
+            for index, document in enumerate(contexts[:2]):
+                original_text = document.metadata.get('original_text', '')
+                # Replace curly brackets, as otherwise problems can be encountered with formatting the prompt
+                original_text = original_text.replace("{", "").replace("}", "")
+                concatenated_contexts += f"{index+1}. {original_text}\n\n"
+
+                source = document.metadata.get('source', '')
+                if source not in seen_sources: 
+                    concatenated_sources += f"{len(seen_sources) + 1}. {source}\n"
+                    seen_sources.add(source)
+
+        else:
+            for index, document in enumerate(contexts):
+                original_text = document.metadata.get('original_text', '')
+                # Replace curly brackets, as otherwise problems can be encountered with formatting the prompt
+                original_text = original_text.replace("{", "").replace("}", "")
+                concatenated_contexts += f"{index+1}. {original_text}\n\n"
+
+                source = document.metadata.get('source', '')
+                if source not in seen_sources: 
+                    concatenated_sources += f"{len(seen_sources) + 1}. {source}\n"
+                    seen_sources.add(source)
+
+        # Check if question mark is at the end of the prompt
+        if not prompt.endswith("?"):
+            prompt = prompt + "?"
+
+        if is_template_question:
+            system_str, system_prompt = get_system_prompt_template()
+            human_str, human_prompt = get_human_prompt(concatenated_contexts, prompt)
+            full_prompt = get_full_prompt(system_prompt, human_prompt)
+            #answer = llm.invoke(full_prompt).content
+        else:
+            system_str, system_prompt = get_system_prompt()
+            human_str, human_prompt = get_human_prompt(concatenated_contexts, prompt)
+            full_prompt = get_full_prompt(system_prompt, human_prompt)
+            #answer = llm.invoke(full_prompt).content
+        
+        return full_prompt, concatenated_sources, template_path, template_source

app/prompts.py

@@ -0,0 +1,39 @@
+from typing import Tuple
+from typing import List
+from langchain.prompts.chat import (
+    ChatPromptTemplate,
+    SystemMessagePromptTemplate,
+    HumanMessagePromptTemplate,
+)
+from langchain_core.messages import BaseMessage
+from langchain.docstore.document import Document
+
+"""
+Python file for getting the prompts and the respective templates.
+"""
+
+def get_system_prompt() -> Tuple[str, SystemMessagePromptTemplate]:
+
+    prompt_str = """You are an expert in information security, especially for ISO 27001 certifications. Answer the following question as truthfully as possible, using the provided context and not prior knowledge. If the answer is not contained within the context or the question is not related to the topic of information security or ISO 27001 or the question is not written in English, respond with 'I am sorry. I do not have knowledge on that topic'. Write a maximum of 400 words."""
+    template = SystemMessagePromptTemplate.from_template(prompt_str)
+    return prompt_str, template
+
+def get_system_prompt_template() -> Tuple[str, SystemMessagePromptTemplate]:
+
+    prompt_str = f"""Answer the following question with that you can provide a template to the user and say that it is attached to this message. After that end your answer."""
+    template = SystemMessagePromptTemplate.from_template(prompt_str)
+
+    return prompt_str, template
+
+def get_human_prompt(contexts: List[str], question: str) -> Tuple[str, HumanMessagePromptTemplate]:
+
+    prompt_str = f"""Question: {question} \n Context: {contexts}"""
+    template = HumanMessagePromptTemplate.from_template(prompt_str)
+    return prompt_str, template
+
+def get_full_prompt(system_prompt: SystemMessagePromptTemplate, human_prompt: HumanMessagePromptTemplate) -> List[BaseMessage]:
+
+    full_prompt = ChatPromptTemplate.from_messages([system_prompt, human_prompt])
+    prompt_messages = full_prompt.format_prompt().to_messages()
+
+    return prompt_messages

chroma/28f27476-0d16-464f-b0fa-5ebbcf277b95/data_level0.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:95042e844cfb77b20e578cf65635282a99d7c4dd20e589ac062f38bc389f8e58
+size 4236000

chroma/28f27476-0d16-464f-b0fa-5ebbcf277b95/header.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fcc596bc1909f7cc610d5839236c90513b4fbad06776c253fa1b21bfd712e940
+size 100

chroma/28f27476-0d16-464f-b0fa-5ebbcf277b95/length.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fc19b1997119425765295aeab72d76faa6927d4f83985d328c26f20468d6cc76
+size 4000

chroma/28f27476-0d16-464f-b0fa-5ebbcf277b95/link_lists.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
+size 0

chroma/chroma.sqlite3

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:491712cb0724ea8cd14691afba9f5c0ae3f07b780ec11e0a383b24c6cd711fe6
+size 10010624

index_preparation/build_index.ipynb

@@ -0,0 +1,250 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Notebook for creating/updating the dense and sparse indices"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from ipynb.fs.defs.preprocess_data import preprocess_data\n",
+    "from ipynb.fs.defs.preprocess_data import get_documents_from_files\n",
+    "from ipynb.fs.defs.preprocess_data import split_docs\n",
+    "from ipynb.fs.defs.preprocess_data import clean_and_process_chunked_documents\n",
+    "from ipynb.fs.defs.preprocess_data import store_documents\n",
+    "import chromadb\n",
+    "from langchain.vectorstores import Chroma\n",
+    "from langchain.docstore.document import Document\n",
+    "from typing import List\n",
+    "import os\n",
+    "\n",
+    "\n",
+    "def build_or_update_index_vector_db(documents: List[Document], embeddings, collection_name: str, dist_function: str, collection_metadata: dict):\n",
+    "    '''\n",
+    "    Builds the index vector DB from documents with the specified embeddings and collection_name\n",
+    "    If it already exists, updates the index with the new documents\n",
+    "    '''\n",
+    "    new_client = chromadb.PersistentClient(path=os.environ.get(\"CHROMA_PATH\"))\n",
+    "\n",
+    "    print(\"Starting to build index for: \", collection_metadata)\n",
+    "\n",
+    "    # Check if collection already exists\n",
+    "    collection_exists = True\n",
+    "    try:\n",
+    "        collection = new_client.get_collection(collection_name)\n",
+    "    except ValueError as e:\n",
+    "        collection_exists = False\n",
+    "\n",
+    "    if not collection_exists:\n",
+    "        print(\"Collection is new\")\n",
+    "        # If collection does not exist, create it\n",
+    "        collection = new_client.create_collection(collection_name)\n",
+    "        # Each document needs an ID\n",
+    "        ids = [str(i) for i in range(1, len(documents) + 1)]\n",
+    "\n",
+    "        # Store the text of the document and metadata separately in order to insert it into Chroma\n",
+    "        texts = []\n",
+    "        metadata_docs = []\n",
+    "        for document in documents:\n",
+    "            texts.append(document.page_content)\n",
+    "            metadata_docs.append(document.metadata)\n",
+    "\n",
+    "        # Add them in batches (otherwise Chroma error)\n",
+    "        for start_idx in range(0, len(embeddings), 1000):\n",
+    "            end_idx = start_idx + 1000\n",
+    "            # Ensure not to go out of bounds\n",
+    "            embeddings_batch = embeddings[start_idx : min(end_idx, len(embeddings))]\n",
+    "            texts_batch = texts[start_idx : min(end_idx, len(embeddings))]\n",
+    "            ids_batch = ids[start_idx : min(end_idx, len(embeddings))]\n",
+    "            metadatas_batch = metadata_docs[start_idx : min(end_idx, len(embeddings))]\n",
+    "\n",
+    "            collection.add(embeddings=embeddings_batch, documents=texts_batch, ids=ids_batch, metadatas=metadatas_batch)\n",
+    "            print(f\"Added embeddings from {start_idx} to {min(end_idx, len(embeddings))-1}\")\n",
+    "\n",
+    "        vectordb = Chroma(\n",
+    "            client=new_client,\n",
+    "            collection_name=collection_name,\n",
+    "            collection_metadata={\n",
+    "                \"embedding_model_provider\": collection_metadata[\"embedding_model_provider\"],\n",
+    "                \"embedding_model_name\": collection_metadata[\"embedding_model_name\"],\n",
+    "                \"chunk_size\": collection_metadata[\"chunk_size\"],\n",
+    "                \"chunk_overlap\": collection_metadata[\"chunk_overlap\"],\n",
+    "                \"hnsw:space\": dist_function,  # either \"l2\" or \"ip\" or \"cosine\"\n",
+    "            },\n",
+    "        )\n",
+    "        print(f\"Collection {collection_name} successfully created.\")\n",
+    "        print(\"There are\", vectordb._collection.count(), \"entries in the collection.\")\n",
+    "\n",
+    "        return new_client, vectordb\n",
+    "\n",
+    "    else:\n",
+    "        print(\"Collection already exists\")\n",
+    "        vectordb = Chroma(client=new_client, collection_name=collection_name)\n",
+    "\n",
+    "        collection_count = vectordb._collection.count()\n",
+    "        print(f\"There are {collection_count} entries in the collection {collection_name} prior to updating.\")\n",
+    "\n",
+    "        # Continue the IDs from the last ID\n",
+    "        ids = [str(i) for i in range(collection_count + 1, collection_count + len(documents) + 1)]\n",
+    "        # Store the text of the document and metadata separately in order to insert it into Chroma\n",
+    "        texts = []\n",
+    "        metadata_docs = []\n",
+    "        for document in documents:\n",
+    "            texts.append(document.page_content)\n",
+    "            metadata_docs.append(document.metadata)\n",
+    "\n",
+    "        # Add them in batches (otherwise Chroma error)\n",
+    "        for start_idx in range(0, len(embeddings), 1000):\n",
+    "            end_idx = start_idx + 1000\n",
+    "            # Ensure not to go out of bounds\n",
+    "            embeddings_batch = embeddings[start_idx : min(end_idx, len(embeddings))]\n",
+    "            texts_batch = texts[start_idx : min(end_idx, len(embeddings))]\n",
+    "            ids_batch = ids[start_idx : min(end_idx, len(embeddings))]\n",
+    "            metadatas_batch = metadata_docs[start_idx : min(end_idx, len(embeddings))]\n",
+    "\n",
+    "            collection.add(embeddings=embeddings_batch, documents=texts_batch, ids=ids_batch, metadatas=metadatas_batch)\n",
+    "            print(f\"Added embeddings from {start_idx} to {min(end_idx, len(embeddings))-1}\")\n",
+    "\n",
+    "        collection_count = vectordb._collection.count()\n",
+    "        print(f\"There are {collection_count} entries in the collection {collection_name} after updating.\")\n",
+    "        return new_client, 0"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "chunk_size = 1536\n",
+    "chunk_overlap = 264\n",
+    "# If update is needed, set to False\n",
+    "all_docs = True\n",
+    "\n",
+    "documents, embedding_model, embeddings = preprocess_data(chunk_size, chunk_overlap, all_docs)\n",
+    "collection_name = \"ISO_27001_Collection\"\n",
+    "collection_metadata = {\n",
+    "\"embedding_model_provider\": \"Fine-tuned\",\n",
+    "\"embedding_model_name\": \"finetuned-BGE-large-ISO-27001\",\n",
+    "\"chunk_size\": str(chunk_size),\n",
+    "\"chunk_overlap\": str(chunk_overlap),\n",
+    "}\n",
+    "\n",
+    "build_or_update_index_vector_db(documents, embeddings, collection_name, \"l2\", collection_metadata)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def store_documents_for_sparse_retrieval(chunk_size: int, chunk_overlap: int):\n",
+    "    \"\"\"\n",
+    "    Stores the documents for sparse retrieval in a basic text file\n",
+    "    \"\"\"\n",
+    "    documents = get_documents_from_files(True)\n",
+    "    chunked_documents = split_docs(documents, chunk_size=chunk_size, chunk_overlap=chunk_overlap)\n",
+    "    chunked_cleaned_documents = clean_and_process_chunked_documents(chunked_documents)\n",
+    "\n",
+    "    store_documents(chunked_cleaned_documents, f\"./../sparse_index/sparse_1536_264\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Create the actual sparse index\n",
+    "store_documents_for_sparse_retrieval(chunk_size, chunk_overlap)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Helper methods for Chroma"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Returns the vectorDB based on the collection name if it exists\n",
+    "def get_index_vector_db(collection_name: str):\n",
+    "    new_client = chromadb.PersistentClient(path=os.environ.get(\"CHROMA_PATH\"))\n",
+    "\n",
+    "    # Check if collection already exists\n",
+    "    collection_exists = True\n",
+    "    try:\n",
+    "        new_client.get_collection(collection_name)\n",
+    "    except ValueError as e:\n",
+    "        collection_exists = False\n",
+    "\n",
+    "    if not collection_exists:\n",
+    "        raise Exception(\"Error, raised exception: Collection does not exist.\")\n",
+    "    else:\n",
+    "        vectordb = Chroma(client=new_client, collection_name=collection_name)\n",
+    "\n",
+    "        return new_client, vectordb"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def delete_collection(collection_name: str):\n",
+    "    new_client = chromadb.PersistentClient(path=os.environ.get(\"CHROMA_PATH\"))\n",
+    "\n",
+    "    try:\n",
+    "        new_client.delete_collection(collection_name)\n",
+    "    except ValueError as e:\n",
+    "        print(\"Collection could not be deleted.\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def return_collections():\n",
+    "    new_client = chromadb.PersistentClient(path=os.environ.get(\"CHROMA_PATH\"))\n",
+    "    collections = new_client.list_collections()\n",
+    "    return collections"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "venv",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.5"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

index_preparation/create_QA_set_documents.ipynb

@@ -0,0 +1,84 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Notebook for creating the documents based on the curated QA pair dataset"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from ipynb.fs.defs.preprocess_data import store_documents\n",
+    "from langchain.docstore.document import Document\n",
+    "import json\n",
+    "\n",
+    "# Load QA dataset\n",
+    "with open(\"./../input_data/QA_dataset/golden_qa_set.json\", 'r') as file:\n",
+    "    golden_qa_set = json.load(file)\n",
+    "\n",
+    "# Remove duplicate answers (Kersten + Secondary Literature) and template answers\n",
+    "indices_to_remove = list(range(102, 121)) + list(range(122, 133)) + list(range(134, 157))\n",
+    "indices_to_remove = sorted(set(indices_to_remove), reverse=True)\n",
+    "for index in indices_to_remove:\n",
+    "    del golden_qa_set['qa_set'][index]\n",
+    "\n",
+    "question_set = [qa['question'] for qa in golden_qa_set['qa_set']]\n",
+    "golden_answer_set = [qa['golden_answer'] for qa in golden_qa_set['qa_set']]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Create one document for each question\n",
+    "all_qa_dataset_documents = []\n",
+    "for q, a in zip(question_set, golden_answer_set):\n",
+    "\n",
+    "    document = Document(\n",
+    "    page_content=f\"{q} \\n {a}\", \n",
+    "    metadata={\n",
+    "        \"source\": \"QA Dataset\",\n",
+    "        \"title\": \"QA Dataset\"\n",
+    "    })\n",
+    "    all_qa_dataset_documents.append(document)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "store_documents(all_qa_dataset_documents, \"./../input_data/QA_dataset/all_documents\")"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "venv",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.8.5"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

index_preparation/create_pdf_documents.ipynb

@@ -0,0 +1,139 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Notebook for updating the PDF document"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from ipynb.fs.defs.preprocess_data import store_documents\n",
+    "from ipynb.fs.defs.preprocess_data import load_documents\n",
+    "from langchain.docstore.document import Document\n",
+    "import pypdfium2 as pdfium\n",
+    "import cv2\n",
+    "import os\n",
+    "import pytesseract\n",
+    "from typing import List\n",
+    "import shutil\n",
+    "\n",
+    "pytesseract_path = os.environ.get(\"TESSERACT_PATH\")\n",
+    "pytesseract.pytesseract.tesseract_cmd = pytesseract_path\n",
+    "\n",
+    "\n",
+    "def update_pdf_documents() -> List[Document]:\n",
+    "    \"\"\"\n",
+    "    Method for processing and updating documents based on the PDFs stored in input_data/PDF/documents. For that the PDFs, that were not processed yet, are converted to images and then transformed to texts. For each PDF one document is then created with all text from all pages. In the end the filename is changed, so that it is clear that it was already processed.\n",
+    "    \"\"\"\n",
+    "\n",
+    "    # List for either all documents or only new ones\n",
+    "    documents_PDF = []\n",
+    "    # List for all documents\n",
+    "    already_processed_documents = load_documents(\"./../input_data/PDF/documents/all_documents\")\n",
+    "\n",
+    "    PDF_images_path = \"./../input_data/PDF/PDF_Images\"\n",
+    "    directory_path = \"./../input_data/PDF/files\"\n",
+    "\n",
+    "    # Go through each PDF file in the directory\n",
+    "    for file in os.listdir(directory_path):\n",
+    "        if \"Tesseract_processed\" not in file:\n",
+    "            file_path = os.path.join(directory_path, file)\n",
+    "            pdf = pdfium.PdfDocument(file_path)\n",
+    "            n_pages = len(pdf)\n",
+    "            # Create directory to store the image\n",
+    "            os.makedirs(PDF_images_path + f\"/{file}\")\n",
+    "            complete_text = \"\"\n",
+    "            # Go through each page of the PDF and save the according image\n",
+    "            for page_number in range(n_pages):\n",
+    "                page = pdf.get_page(page_number)\n",
+    "                pil_image = page.render(\n",
+    "                    scale=300 / 72,\n",
+    "                    rotation=0,\n",
+    "                    crop=(0, 0, 0, 0),\n",
+    "                ).to_pil()\n",
+    "                pil_image_path = PDF_images_path + f\"/{file}/image_{page_number+1}.png\"\n",
+    "                pil_image.save(pil_image_path)\n",
+    "                img = cv2.imread(pil_image_path)\n",
+    "                # Convert image to grayscale\n",
+    "                gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)\n",
+    "                # Apply threshold to convert to binary image\n",
+    "                threshold_img = cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)[1]\n",
+    "                # Pass the image through pytesseract and add the text to the whole document text\n",
+    "                complete_text += pytesseract.image_to_string(threshold_img) + \"\\n\"\n",
+    "                # Remove the image as it is already processed\n",
+    "                os.remove(pil_image_path)\n",
+    "\n",
+    "                file_name_without_pdf = file\n",
+    "                if file.endswith(\".pdf\"):\n",
+    "                    file_name_without_pdf = file[:-4]\n",
+    "            # Create a document based on the whole text and metadata\n",
+    "            document_PDF = Document(page_content=complete_text, metadata={\"source\": file, \"title\": file_name_without_pdf})\n",
+    "            documents_PDF.append(document_PDF)\n",
+    "            already_processed_documents.append(document_PDF)\n",
+    "\n",
+    "            # Change the filename, so that in future calls the PDF is not processed again\n",
+    "            new_filename = file.replace(\".pdf\", \"_Tesseract_processed.pdf\")\n",
+    "            new_pdf_path = os.path.join(directory_path, new_filename)\n",
+    "            print(new_pdf_path)\n",
+    "            pdf.close()\n",
+    "            os.rename(file_path, new_pdf_path)\n",
+    "\n",
+    "    # Store docs if new documents were processed\n",
+    "    if len(documents_PDF) > 0:\n",
+    "        # Store all documents, including the new ones\n",
+    "        store_documents(already_processed_documents, \"./../input_data/PDF/documents/all_documents\")\n",
+    "        # Store the new documents\n",
+    "        store_documents(documents_PDF, \"./../input_data/PDF/documents/new_documents\")\n",
+    "\n",
+    "    # Delete the empty folders inside the images folder\n",
+    "    target_dir = \"./../input_data/PDF/PDF_images\"\n",
+    "\n",
+    "    # Check if the target directory exists to avoid errors\n",
+    "    if os.path.exists(target_dir):\n",
+    "        # List all the items in the directory\n",
+    "        for item in os.listdir(target_dir):\n",
+    "            item_path = os.path.join(target_dir, item)\n",
+    "            if os.path.isdir(item_path):\n",
+    "                # Use shutil.rmtree to delete the directory and all its contents\n",
+    "                shutil.rmtree(item_path)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Uncomment update needed because of new unprocessed files\n",
+    "# update_pdf_documents()"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "venv",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.8.5"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

index_preparation/create_template_documents.ipynb

@@ -0,0 +1,140 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Notebook for creating the template documents"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from ipynb.fs.defs.preprocess_data import get_template_documents\n",
+    "from ipynb.fs.defs.preprocess_data import store_documents\n",
+    "from langchain.docstore.document import Document\n",
+    "import os\n",
+    "import shutil\n",
+    "\n",
+    "# Load all already existing documents and store paths of new documents to be processed\n",
+    "all_template_documents = get_template_documents(True)\n",
+    "template_paths = os.listdir(\"./../input_data/Templates/template_files/new\")\n",
+    "\n",
+    "print(all_template_documents)\n",
+    "print(\"\\n\")\n",
+    "print(template_paths)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Manually create the documents for each template\n",
+    "full_path = \"./../input_data/Templates/template_files/processed\"\n",
+    "\n",
+    "template_document_1 = Document(\n",
+    "    page_content=f\"You can find a possible template for the backup policy from the Annex A of ISO 27001 attached to this message. It contains pre-written texts for purpose, scope, content and more for the backup policy.\", \n",
+    "    metadata={\n",
+    "        \"template_path\": full_path + \"/\" + template_paths[0], \n",
+    "        \"source\": template_paths[0],\n",
+    "    })\n",
+    "\n",
+    "template_document_2 = Document(\n",
+    "    page_content=f\"You can find a possible template for the change management policy from the Annex A of ISO 27001 attached to this message. It contains pre-written texts for purpose, scope, content, procedures, risk management and more for the change management policy.\", \n",
+    "    metadata={\n",
+    "        \"template_path\": full_path + \"/\" + template_paths[1], \n",
+    "        \"source\": template_paths[1],\n",
+    "    })\n",
+    "\n",
+    "\n",
+    "template_document_3 = Document(\n",
+    "    page_content=f\"You can find a possible template for the encryption policy from the Annex A of ISO 27001 attached to this message. It contains pre-written texts for purpose, scope, content and more for the encryption policy.\", \n",
+    "    metadata={\n",
+    "        \"template_path\": full_path + \"/\" + template_paths[2], \n",
+    "        \"source\": template_paths[2],\n",
+    "    })\n",
+    "\n",
+    "\n",
+    "template_document_4 = Document(\n",
+    "    page_content=f\"You can find a possible template for a checklist for all ISO-27001 controls (Version 2013) attached to this message. It contains a simple checklist for the ISO 27001 controls 5 to 18.\", \n",
+    "    metadata={\n",
+    "        \"template_path\": full_path + \"/\" + template_paths[3], \n",
+    "        \"source\": template_paths[3],\n",
+    "    })\n",
+    "\n",
+    "template_document_5 = Document(\n",
+    "    page_content=f\"You can find a possible template for a risk assessment needed for the ISO-27001 certification attached to this message. It contains a simple checklist of selected controls for which a risk assessment is needed.\", \n",
+    "    metadata={\n",
+    "        \"template_path\": full_path + \"/\" + template_paths[4], \n",
+    "        \"source\": template_paths[4],\n",
+    "    })\n",
+    "\n",
+    "template_document_5 = Document(\n",
+    "    page_content=f\"You can find a possible template for a risk assessment needed for the ISO-27001 certification attached to this message. It contains a simple checklist of selected controls for which a risk assessment is needed.\", \n",
+    "    metadata={\n",
+    "        \"template_path\": full_path + \"/\" + template_paths[4], \n",
+    "        \"source\": template_paths[4],\n",
+    "    })\n",
+    "\n",
+    "new_template_documents = []\n",
+    "new_template_documents.append(template_document_1)\n",
+    "new_template_documents.append(template_document_2)\n",
+    "new_template_documents.append(template_document_3)\n",
+    "new_template_documents.append(template_document_4)\n",
+    "new_template_documents.append(template_document_5)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Store the new templates and all templates\n",
+    "store_documents(new_template_documents, \"./../input_data/Templates/documents/new_documents\")\n",
+    "\n",
+    "all_template_documents.extend(new_template_documents)\n",
+    "store_documents(new_template_documents, \"./../input_data/Templates/documents/all_documents\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Move new templates to processed templates\n",
+    "for path in template_paths:\n",
+    "    source_file = f\"./../input_data/Templates/template_files/new/{path}\"\n",
+    "    destination_folder = \"./../input_data/Templates/template_files/processed\"\n",
+    "    shutil.move(source_file, destination_folder)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "venv",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.5"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

index_preparation/create_web_documents.ipynb

@@ -0,0 +1,176 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Notebook for cleaning, creating and updating the web documents\n",
+    "\n",
+    "First import the documents from the uncleaned URLs and store the text into separate files. Then manually clean files and update it. After that, get all already cleaned documents and store both, only the new documents and all documents (old cleaned + new cleaned)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from ipynb.fs.defs.preprocess_data import get_web_documents\n",
+    "from ipynb.fs.defs.preprocess_data import store_documents\n",
+    "from langchain.docstore.document import Document\n",
+    "from langchain.document_loaders import AsyncHtmlLoader\n",
+    "from langchain.document_transformers import Html2TextTransformer\n",
+    "from typing import List\n",
+    "\n",
+    "def get_web_documents_for_cleaning() -> List[Document]:\n",
+    "    \"\"\"\n",
+    "    Method for returning documents based on the URLs. Looks at the .txt file with all uncleaned urls and uses the AsyncHTMLoader and HTML2TextTransformer to get the texts.\n",
+    "    \"\"\"\n",
+    "    directory_path_web = \"./../input_data/Web/URLs/uncleaned_urls.txt\"\n",
+    "\n",
+    "    imported_urls = []\n",
+    "    with open(directory_path_web, \"r\") as file:\n",
+    "        for line in file:\n",
+    "            imported_urls.append(line.strip())\n",
+    "\n",
+    "    loader_web = AsyncHtmlLoader(imported_urls)\n",
+    "    documents_web = loader_web.load()\n",
+    "\n",
+    "    html2text = Html2TextTransformer()\n",
+    "    documents_web_transformed = html2text.transform_documents(documents_web)\n",
+    "    print(\"Number of documents: \" + str(len(documents_web_transformed)) + \"\\n\")\n",
+    "\n",
+    "    return documents_web_transformed\n",
+    "\n",
+    "documents = get_web_documents_for_cleaning()\n",
+    "already_cleaned_documents = get_web_documents(True)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Loop over the array and store each string in a separate txt file\n",
+    "counter = 1\n",
+    "for doc in documents:\n",
+    "    # Specify the file name for each string (e.g., file0.txt, file1.txt, ...)\n",
+    "    file_name = f\"file_{counter}.txt\"\n",
+    "    counter += 1\n",
+    "    \n",
+    "    # Open the file in write mode\n",
+    "    with open(file_name, 'w', encoding='utf-8') as file:\n",
+    "        # Write the string to the file\n",
+    "        file.write(doc.page_content)\n",
+    "\n",
+    "    print(f'The string has been successfully stored in {file_name}.')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# NOW MANUALLY CLEAN"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "cleaned_texts = []\n",
+    "\n",
+    "counter = 1\n",
+    "for doc in documents:\n",
+    "    # Specify the file name for each string (e.g., file0.txt, file1.txt, ...)\n",
+    "    file_name = f\"file_{counter}.txt\"\n",
+    "    counter += 1\n",
+    "    \n",
+    "    # Open the file in write mode\n",
+    "    with open(file_name, 'r', encoding='utf-8') as file:\n",
+    "        # Write the string to the file\n",
+    "        text = file.read()\n",
+    "        cleaned_texts.append(text)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Set the new cleaned texts\n",
+    "if len(documents) == len(cleaned_texts):\n",
+    "    for i in range(len(documents)):\n",
+    "        documents[i].page_content = cleaned_texts[i]\n",
+    "else:\n",
+    "    raise Exception(\"Error.\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Store only the new documents and all documents\n",
+    "store_documents(documents, \"./../input_data/Web/documents/new_documents\")\n",
+    "\n",
+    "already_cleaned_documents.extend(documents)\n",
+    "store_documents(already_cleaned_documents, \"./../input_data/Web/documents/all_documents\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Update the URLs list for cleaned and uncleaned\n",
+    "uncleaned_url_file_path = \"./../input_data/Web/URLs/uncleaned_urls.txt\"\n",
+    "cleaned_url_file_path = \"./../input_data/Web/URLs/cleaned_urls.txt\"\n",
+    "\n",
+    "# Read URLs from the source file and store them in a list\n",
+    "with open(uncleaned_url_file_path, \"r\") as source_file:\n",
+    "    urls = source_file.readlines()\n",
+    "\n",
+    "# Open the destination file in append mode and write the URLs to it\n",
+    "with open(cleaned_url_file_path, \"a\") as destination_file:\n",
+    "    destination_file.writelines(urls)\n",
+    "\n",
+    "# Remove the URLs from the source file\n",
+    "with open(uncleaned_url_file_path, \"w\") as source_file:\n",
+    "    source_file.write(\"\")\n",
+    "\n",
+    "# Print moved urls\n",
+    "for url in urls:\n",
+    "    print(\"Moved URL:\", url.strip())"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "venv",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.8.5"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

index_preparation/preprocess_data.ipynb

@@ -0,0 +1,229 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Notebook for processing the text data (chunking, cleaning, embeddings)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import json\n",
+    "from typing import Iterable\n",
+    "from langchain.docstore.document import Document\n",
+    "from typing import List\n",
+    "\n",
+    "# Helper methods for storing and loading already generated documents\n",
+    "def store_documents(documents, file_path: str) -> None:\n",
+    "    with open(file_path, \"w\") as jsonl_file:\n",
+    "        for doc in documents:\n",
+    "            jsonl_file.write(doc.json() + \"\\n\")\n",
+    "\n",
+    "\n",
+    "def load_documents(file_path: str) -> List[Document]:\n",
+    "    documents = []\n",
+    "    with open(file_path, \"r\") as jsonl_file:\n",
+    "        for line in jsonl_file:\n",
+    "            data = json.loads(line)\n",
+    "            obj = Document(**data)\n",
+    "            documents.append(obj)\n",
+    "    return documents"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def get_pdf_documents(all_docs: bool):\n",
+    "    \"\"\"\n",
+    "    Method for returning the documents of the PDFs. Processing and updating takes place in update_pdf_documents.\n",
+    "    all_docs parameter defines whether to load all documents or only new ones. Only new ones can be used if the index is already build and new documents should be added.\n",
+    "    \"\"\"\n",
+    "    pdf_documents = []\n",
+    "    if all_docs:\n",
+    "        pdf_documents = load_documents(\"./../input_data/PDF/documents/all_documents\")\n",
+    "    else:\n",
+    "        pdf_documents = load_documents(\"./../input_data/PDF/documents/new_documents\")\n",
+    "\n",
+    "    return pdf_documents\n",
+    "\n",
+    "def get_web_documents(all_docs: bool) -> List[Document]:\n",
+    "    \"\"\"\n",
+    "    Method for returning the already processed documents. FIRST need to call get_web_docs_for_cleaning and clean manually. As it is a manual cleaning process, the methods are need to be called asynchronously.\n",
+    "    \"\"\"\n",
+    "    web_documents = []\n",
+    "    if all_docs:\n",
+    "        web_documents = load_documents(\"./../input_data/Web/documents/all_documents\")\n",
+    "    else:\n",
+    "        web_documents = load_documents(\"./../input_data/Web/documents/new_documents\")\n",
+    "\n",
+    "    return web_documents\n",
+    "\n",
+    "def get_template_documents(all_docs: bool) -> List[Document]:\n",
+    "    \"\"\"\n",
+    "    Method for returning the documents of the templates.\n",
+    "    \"\"\"\n",
+    "    template_documents = []\n",
+    "    if all_docs:\n",
+    "        template_documents = load_documents(\"./../input_data/Templates/documents/all_documents\")\n",
+    "    else:\n",
+    "        template_documents = load_documents(\"./../input_data/Templates/documents/new_documents\")\n",
+    "\n",
+    "    return template_documents\n",
+    "\n",
+    "def get_dataset_documents() -> List[Document]:\n",
+    "    \"\"\"\n",
+    "    Method for returning the documents of the templates.\n",
+    "    \"\"\"\n",
+    "    template_documents = []\n",
+    "    template_documents = load_documents(\"./../input_data/QA_dataset/all_documents\")\n",
+    "\n",
+    "    return template_documents"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def get_documents_from_files(all_docs: bool):\n",
+    "    \"\"\"\n",
+    "    Gets documents from all document types.\n",
+    "    \"\"\"\n",
+    "    documents_all = []\n",
+    "    documents_PDF = get_pdf_documents(all_docs)\n",
+    "    document_web = get_web_documents(all_docs)\n",
+    "    document_template = get_template_documents(all_docs)\n",
+    "    document_dataset = get_dataset_documents()\n",
+    "    \n",
+    "    documents_all.extend(documents_PDF)\n",
+    "    documents_all.extend(document_web)\n",
+    "    documents_all.extend(document_template)\n",
+    "    documents_all.extend(document_dataset)\n",
+    "    \n",
+    "    print(\"Number of documents: \" + str(len(documents_all)) + \"\\n\")\n",
+    "    return documents_all"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
+    "\n",
+    "def split_docs(documents: List[Document], chunk_size: int, chunk_overlap: int):\n",
+    "\n",
+    "    text_splitter = RecursiveCharacterTextSplitter(chunk_size=chunk_size, chunk_overlap=chunk_overlap, separators=[\" \"])\n",
+    "    chunkedDocuments = text_splitter.split_documents(documents)\n",
+    "    return chunkedDocuments"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import re\n",
+    "\n",
+    "def clean_text(text: str) -> str:\n",
+    "    # Replace multiple whitespaces (except newlines) with a single space\n",
+    "    text = re.sub(r\"(?!\\n)\\s+\", \" \", text)\n",
+    "    # Replace multiple newlines with a single newline\n",
+    "    text = re.sub(r\"\\n+\", \"\\n\", text)\n",
+    "    # Remove leading and trailing whitespace\n",
+    "    text = text.strip()\n",
+    "    return text\n",
+    "\n",
+    "def clean_and_process_chunked_documents(chunkedDocuments: List[Document]) -> List[Document]:\n",
+    "    counter = 1\n",
+    "    for i in chunkedDocuments:\n",
+    "        i.page_content = clean_text(i.page_content)\n",
+    "        i.metadata[\"original_text\"] = i.page_content\n",
+    "        i.metadata[\"doc_ID\"] = counter\n",
+    "        counter += 1\n",
+    "\n",
+    "        i.page_content = i.page_content.lower()       \n",
+    "\n",
+    "    return chunkedDocuments"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.embeddings import HuggingFaceEmbeddings\n",
+    "\n",
+    "def get_embedding_model():\n",
+    "    path = \"Basti8499/bge-large-en-v1.5-ISO-27001\"\n",
+    "    model = HuggingFaceEmbeddings(model_name=path)\n",
+    "    return model"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def create_embedding_vectors(embedding_model, documents: List[Document]):\n",
+    "    texts = []\n",
+    "    for document in documents:\n",
+    "        texts.append(document.page_content)\n",
+    "\n",
+    "    embeddings = embedding_model.embed_documents(texts)\n",
+    "\n",
+    "    return embeddings"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def preprocess_data(chunk_size: int, chunk_overlap: int, all_docs: bool):\n",
+    "    documents = get_documents_from_files(all_docs)\n",
+    "    chunked_documents = split_docs(documents, chunk_size=chunk_size, chunk_overlap=chunk_overlap)\n",
+    "    chunked_cleaned_documents = clean_and_process_chunked_documents(chunked_documents)\n",
+    "    embedding_model = get_embedding_model()\n",
+    "    embeddings = create_embedding_vectors(embedding_model, chunked_cleaned_documents)\n",
+    "\n",
+    "    return chunked_cleaned_documents, embedding_model, embeddings"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "venv",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.5"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

init_embedding_model.py

@@ -0,0 +1,4 @@
+from langchain_community.embeddings import HuggingFaceEmbeddings
+
+path = "Basti8499/bge-large-en-v1.5-ISO-27001"
+model = HuggingFaceEmbeddings(model_name=path)

input_data/Templates/documents/all_documents

@@ -0,0 +1,5 @@
+{"page_content": "You can find a possible template for the backup policy from the Annex A of ISO 27001 attached to this message. It contains pre-written texts for purpose, scope, content and more for the backup policy.", "metadata": {"template_path": "./../input_data/Templates/template_files/processed/Backup policy.docx", "source": "Backup policy.docx"}, "type": "Document"}
+{"page_content": "You can find a possible template for the change management policy from the Annex A of ISO 27001 attached to this message. It contains pre-written texts for purpose, scope, content, procedures, risk management and more for the change management policy.", "metadata": {"template_path": "./../input_data/Templates/template_files/processed/Change management policy.docx", "source": "Change management policy.docx"}, "type": "Document"}
+{"page_content": "You can find a possible template for the encryption policy from the Annex A of ISO 27001 attached to this message. It contains pre-written texts for purpose, scope, content and more for the encryption policy.", "metadata": {"template_path": "./../input_data/Templates/template_files/processed/Encryption policy.docx", "source": "Encryption policy.docx"}, "type": "Document"}
+{"page_content": "You can find a possible template for a checklist for all ISO-27001 controls (Version 2013) attached to this message. It contains a simple checklist for the ISO 27001 controls 5 to 18.", "metadata": {"template_path": "./../input_data/Templates/template_files/processed/IC-ISO-27001-Controls-Checklist.xlsx", "source": "IC-ISO-27001-Controls-Checklist.xlsx"}, "type": "Document"}
+{"page_content": "You can find a possible template for a risk assessment needed for the ISO-27001 certification attached to this message. It contains a simple checklist of selected controls for which a risk assessment is needed.", "metadata": {"template_path": "./../input_data/Templates/template_files/processed/IC-ISO-27001-Risk-Assessment.xlsx", "source": "IC-ISO-27001-Risk-Assessment.xlsx"}, "type": "Document"}

input_data/Web/URLs/cleaned_urls.txt

@@ -0,0 +1,62 @@
+https://www.vanta.com/resources/who-needs-iso-27001-certification
+https://www.vanta.com/resources/iso-27001-compliance-checklist
+https://www.vanta.com/resources/how-long-does-it-take-to-get-iso-certified
+https://www.strongdm.com/blog/iso-27001-controls
+https://www.itgovernance.eu/blog/en/a-9-step-guide-to-implementing-iso-27001
+https://www.itgovernance.eu/blog/en/benefits-of-iso-27001-certification
+https://www.itgovernance.eu/blog/en/why-are-so-many-organisations-getting-certified-to-iso-27001
+https://www.itgovernance.eu/blog/en/what-you-need-to-know-about-iso-270012022
+https://www.itgovernance.eu/blog/en/how-iso-27001-can-boost-your-cloud-security
+https://www.vanta.com/resources/the-ultimate-iso-27001-guide-powered-by-vanta-and-aprio
+https://www.drata.com/blog/iso-27001-compliance
+https://www.drata.com/blog/iso-27001-risk-assessment
+https://www.drata.com/blog/iso-27001-statement-of-applicability
+https://www.drata.com/blog/ask-an-auditor-demystifying-iso-27001
+https://www.drata.com/blog/iso-27001-vs-iso-27002
+https://www.drata.com/blog/iso-27001-2022-update
+https://www.drata.com/blog/iso-27001-certification-cost
+https://www.dataguard.co.uk/knowledge/iso-27001/2022-version-transition-guide/
+https://www.dataguard.co.uk/knowledge/iso-27001-certification/
+https://www.dataguard.co.uk/knowledge/iso-27001-controls-annex-a/
+https://www.dataguard.co.uk/knowledge/iso-27001/clause-4-1-requirements-of-interested-parties/
+https://www.dataguard.co.uk/knowledge/iso-27001/clause-4-2-needs-and-expectations-of-key-parties/
+https://www.dataguard.co.uk/knowledge/iso-27001/clause-4-3-how-to-determine-the-scope-of-your-isms/
+https://www.dataguard.co.uk/knowledge/iso-27001/clause-4-4-information-security-management-system/
+https://www.dataguard.co.uk/knowledge/iso-27001/clause-5-1-leadership-and-commitment/
+https://www.dataguard.co.uk/knowledge/iso-27001/clause-5-2-information-security-policy/
+https://www.dataguard.co.uk/knowledge/iso-27001/clause-5-3-organisational-roles-responsibilities-and-authorities/
+https://www.dataguard.co.uk/knowledge/iso-27001/clause-6-1-actions-to-address-risks-and-opportunities/
+https://www.dataguard.co.uk/knowledge/iso-27001/clause-6-2-information-security-objectives/
+https://www.british-assessment.co.uk/insights/a-complete-guide-to-iso-iec-270012022/
+https://secureframe.com/blog/iso-27001-2022
+https://www.dataguard.co.uk/blog/iso-27001-risk-treatment-plan-what-you-need-to-know
+https://www.creative-n.com/blog/how-much-does-it-cost-to-implement-iso-27001/
+https://www.secfix.com/post/when-is-an-iso-27001-certification-required
+https://www.dataguard.co.uk/knowledge/iso-27001/clause-7-1-resources-for-isms/
+https://www.dataguard.co.uk/knowledge/iso-27001/clause-7-2-competence/
+https://www.dataguard.co.uk/knowledge/iso-27001/clause-7-3-awareness/
+https://www.dataguard.co.uk/knowledge/iso-27001/clause-7-4-communication/
+https://www.dataguard.co.uk/knowledge/iso-27001/clause-7-5-documented-information/
+https://www.dataguard.co.uk/knowledge/iso-27001/clause-8-1-operational-planning-and-control/
+https://www.dataguard.co.uk/knowledge/iso-27001/clause-8-2-information-security-risk-assessment/
+https://www.dataguard.co.uk/knowledge/iso-27001/clause-8-3-information-security-risk-treatment/
+https://www.dataguard.co.uk/knowledge/iso-27001/clause-9-1-monitoring-measurement-analysis-and-evaluation/
+https://www.dataguard.co.uk/knowledge/iso-27001/clause-9-2-internal-audit/
+https://www.dataguard.co.uk/knowledge/iso-27001/clause-9-3-management-review/
+https://www.dataguard.co.uk/knowledge/iso-27001/clause-10-1-continual-improvement/
+https://www.dataguard.co.uk/knowledge/iso-27001/clause-10-2-nonconformity-and-corrective-action/
+https://www.dataguard.co.uk/blog/iso-27001-annex-a5-information-security-policies
+https://www.dataguard.co.uk/blog/iso-27001-annex-a.6-organisation-information-security/
+https://www.dataguard.co.uk/blog/iso-27001-annex-a.7-human-resource-security/
+https://www.dataguard.co.uk/blog/iso-27001-annex-a.8-asset-management
+https://www.dataguard.co.uk/blog/iso-27001-annex-a.9-access-control/
+https://www.dataguard.co.uk/blog/iso-27001-annex-a.10-cryptography
+https://www.dataguard.co.uk/blog/iso-27001-annex-a.11-physical-and-environmental-security/
+https://www.dataguard.co.uk/blog/iso-27001-annex-a.12-operations-security
+https://www.dataguard.co.uk/blog/iso-27001-annex-a.13-communications-security/
+https://www.dataguard.co.uk/blog/iso-27001-annex-a.14-system-acquisition-development-and-maintenance/
+https://www.dataguard.co.uk/blog/iso-27001-annex-a.15-supplier-relationships/
+https://www.dataguard.co.uk/blog/iso-27001-annex-a.16-information-security-incident-management/
+https://www.dataguard.co.uk/blog/iso-27001-annex-a.17-information-security-aspects-of-business-continuity-management/
+https://www.dataguard.co.uk/blog/iso-27001-annex-a.18-compliance/
+https://reciprocity.com/difference-between-gdpr-and-iso-27001/

setup.sh

@@ -0,0 +1,16 @@
+#!/bin/bash
+python -m venv venv
+source venv/bin/activate
+python -m pip install --upgrade pip
+python -m pip install -r requirements.txt
+python -m pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu121
+echo "Initialization completed."
+
+# Creating all necessary directories at once
+mkdir -p ./inputData/PDF/files \
+         ./inputData/PDF/PDF_images \
+         ./inputData/Templates/template_files/new \
+         ./inputData/Templates/template_files/processed \
+         ./chroma
+
+echo "Directories and necessary files created."