feat: add support for document ingestion and vectorization

Add a new file `constants.py` with a constant `persist_directory` to store the vectorized documents.

In `ingest.py`, import necessary modules and define a `loader` to load documents from a directory. Then, use a `text_splitter` to split the documents into smaller chunks. Next, use an `embedding` to convert the text chunks into vectors. Finally, create a `vectordb` using the `Chroma` vector store and persist it to the `persist_directory`.

constants.py

@@ -0,0 +1 @@
+persist_directory = 'db'

ingest.py

@@ -0,0 +1,27 @@
+from langchain.document_loaders import PyPDFDirectoryLoader
+from langchain.embeddings.openai import OpenAIEmbeddings
+from langchain.text_splitter import RecursiveCharacterTextSplitter
+from langchain.vectorstores import Chroma
+
+from constants import persist_directory
+
+loader = PyPDFDirectoryLoader("docs/")
+documents = loader.load()
+
+text_splitter = RecursiveCharacterTextSplitter(
+    chunk_size=1000,
+    chunk_overlap=200,
+    separators=["\n\n", "\n", ".", "!", ",", " ", ""],
+    keep_separator=True,
+)
+texts = text_splitter.split_documents(documents)
+
+embedding = OpenAIEmbeddings()
+
+vectordb = Chroma.from_documents(
+    documents=texts,
+    embedding=embedding,
+    persist_directory=persist_directory,
+)
+
+vectordb.persist()