clean up. add descriptions

README.md

@@ -11,3 +11,65 @@ license: mit
 ---
 
 Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+
+# Chatbot
+
+This is a chatbot that can answer questions based on a given context. It uses OpenAI embeddings and a language model to generate responses. The chatbot also has the ability to ingest documents and store them in a Pinecone index with a given namespace for later retrieval. The most important feature is to utilize Langchain's self-querying retriever, which automatically creates a filter from the user query on the metadata of stored documents and to execute those filter.
+
+## Setup
+
+To use the chatbot, you will need to provide an OpenAI API key and a Pinecone API key. You can enter these keys in the appropriate fields when prompted. The Pinecone API environment, index, and namespace will also be entered.
+
+## Ingesting Files
+
+If you have files that you would like to ingest, you can do so by selecting "Yes" when prompted to ingest files. You can then upload your files and they will be stored in the given Pinecone index associated with the given namespace for later retrieval. The files can be PDF, doc/docx, txt, or a mixture of them.
+If you have previously ingested files and stored in Pinecone, you may indicate No to 'Ingest file(s)?' and the data in the given Pinecone index/namespace will be used.
+
+## Usage
+
+To use the chatbot, simply enter your question in the text area provided. The chatbot will generate a response based on the context and your question.
+
+
+## Retrieving Documents and self-query filter
+
+To retrieve documents, the chatbot uses a SelfQueryRetriever and a Pinecone index. The chatbot will search the index for documents that are relevant to your question and return them as a list.
+
+Self-query retrieval is described in https://python.langchain.com/en/latest/modules/indexes/retrievers/examples/self_query_retriever.html. In this app, a simple, pre-defined metadata field is provided:
+```
+metadata_field_info = [
+	AttributeInfo(
+		name="author",
+		description="The author of the document/text/piece of context",
+		type="string or list[string]",
+	)
+]
+document_content_description = "Views/opions/proposals suggested by the author on one or more discussion points."
+```
+This assumes the ingested files are named by their authors in order to use the self-querying retriever. Then when the user asks a question about one or more specific authors, a filter will be automatically created and only those files by those authors will be used to generate responses. An example:
+```
+query='space exploration alice view' filter=Comparison(comparator=<Comparator.EQ: 'eq'>, attribute='author', value='Alice')
+query='space exploration bob view' filter=Comparison(comparator=<Comparator.EQ: 'eq'>, attribute='author', value='Bob')
+```
+
+If the filter returns nothing, such as when the user's question is unrelated to the metadata field, or when self-querying is not intended, the app will fallback to the traditional similarity search for answering the question.
+
+This can be easily generalized, e.g., user uploading a metadata field file and description for a particular task. This is subject to future work.
+
+
+## Generating Responses
+
+The chatbot generates responses using a large language model and OpenAI embeddings. It uses the context and your question to generate a response that is relevant and informative.
+
+## Saving Chat History
+
+The chatbot saves chat history in a JSON file. You can view your chat history in chat_history folder. However, as of now, chat history is not used in the Q&A process.
+
+## Screenshot
+* Set up the API keys and Pinecone index/namespace
+![setup](./screenshot/selfquery_setup.png)
+* A first self-querying retrieval
+![query1](./screenshot/selfquery1.png)
+* A second self-querying retrieval
+![query2](./screenshot/selfquery2.png)
+* The fitlers automatically created by the app for the self-querying retrievals
+![filter](./screenshot/filter.png)

app.py

@@ -94,7 +94,7 @@ def ingest(_all_texts, _embeddings, pinecone_index_name, pinecone_namespace):
 	return docsearch
 
 
-def setup_retriever(docsearch, k, llm):
+def setup_retriever(docsearch, llm):
     metadata_field_info = [
         AttributeInfo(
             name="author",
@@ -163,9 +163,7 @@ pinecone_index_name, pinecone_namespace, docsearch_ready, directory_name = init(
 def main(pinecone_index_name, pinecone_namespace, docsearch_ready):
     docsearch_ready = False
     chat_history = []
-    # Get user input of whether to use Pinecone or not
     col1, col2, col3 = st.columns([1, 1, 1])
-    # create the radio buttons and text input fields
     with col1:
         r_ingest = st.radio(
             'Ingest file(s)?', ('Yes', 'No'))
@@ -183,7 +181,6 @@ def main(pinecone_index_name, pinecone_namespace, docsearch_ready):
     with col3:
         pinecone_namespace = st.text_input('Pinecone namespace:')
         temperature = st.slider('Temperature', 0.0, 1.0, 0.1)
-        k_sources = st.slider('# source(s) to print out', 0, 20, 2)
     
     if pinecone_index_name:
         session_name = pinecone_index_name
@@ -204,9 +201,7 @@ def main(pinecone_index_name, pinecone_namespace, docsearch_ready):
                                                  embeddings)
             docsearch_ready = True
     if docsearch_ready:
-        # number of sources (split-documents when ingesting files); default is 4
-        k = min([20, n_texts])
-        retriever = setup_retriever(docsearch, k, llm)
+        retriever = setup_retriever(docsearch, llm)
         CRqa = load_qa_with_sources_chain(llm, chain_type="stuff")
 
         st.title('Chatbot')