Upload 3 files

README.md

@@ -1,13 +1,46 @@
 ---
-title: Search
-emoji: 🐢
-colorFrom: pink
-colorTo: green
+title: Utafiti
+emoji: 📚
+colorFrom: red
+colorTo: blue
 sdk: streamlit
-sdk_version: 1.25.0
+sdk_version: 1.17.0
 app_file: app.py
 pinned: false
-license: gpl
+license: apache-2.0
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+
+# Utafiti (Swahili for Research)
+
+- This is a Streamlit app that uses the sentence-transformers library to find similar sentences in a corpus of text. It is based on the [Sentence-BERT](https://www.sbert.net/) model.
+- The corpus is a collection of research paper titles in the realm of mathematics and computer science. Only the titles are used for purposes of similarity search. The abstracts as thus fetched by matching title to abstract (summary) indices stored as pickle files in the resources directory.
+- For query optimization, the faiss (Facebook AI Similarity Search) library is used to index the corpus. Additionally, clustering using Voronoi cells has been implemented to further optimize the search.
+- Hosted on HuggingFace Spaces at: https://huggingface.co/spaces/eolang/utafiti
+
+## Additonal details include:
+    - Encoding size: 768
+    - Corpus size: 28,000
+    - Library: Sentence-BERT (Sentence Transformers), Faiss, Torch, NumPy, Streamlit
+    - Model: multi-qa-MiniLM-L6-cos-v1
+
+### Possible Fixes (*This is a hobby project, so no promises* :-)
+1. Inlculssion of direct links to Axriv, Papers with Code and Google Scholar links for the papers
+2. An API of some sort to allow for easy integration with other projects (maybe in the infamous version 2.0)
+3. A more robust search engine (maybe in the infamous version 2.0)
+
+
+### Random things to note
+- I'm not running this on Streamlit Sharing, due to an issue with the fails-CPU library. **May** migrate it in case I find a fix.
+- I'm not the best developer, so the code is not the best. I'm still learning and I'm open to suggestions and improvements.
+- I'm not a professional mathematician &/ researcher &/ scientist, so the corpus, algorithms, etc. may not be the best. I'm still learning and I'm open to suggestions and improvements.
+
+### Acknowledgements
+- [Sentence-BERT](https://www.sbert.net/)
+- [multi-qa-MiniLM-L6-cos-v1]((https://huggingface.co/sentence-transformers/multi-qa-MiniLM-L6-cos-v1))
+
+
+## For feedback and suggestions, please contact me on:
+- [LinkedIn](https://www.linkedin.com/in/eolang/)
+
+Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference

app.py

@@ -0,0 +1,85 @@
+import pickle
+import json
+import streamlit as st
+import pandas as pd
+
+
+#load files/resources using pickle
+with open('resources/titles.pkl', 'rb') as f:
+    titles = pickle.load(f)
+
+with open('resources/embeddings.pkl', 'rb') as f:
+    embeddings = pickle.load(f)
+
+with open('resources/authors.pkl', 'rb') as f:
+    authors = pickle.load(f)
+
+with open('resources/years.pkl', 'rb') as f:
+    years = pickle.load(f)
+
+with open('resources/summary.pkl', 'rb') as f:
+    summary = pickle.load(f)
+
+with open('resources/titles_urls.pkl', 'rb') as f:
+    title_urls = pickle.load(f)
+
+
+index = pickle.load(open('resources/index.sav' , 'rb'))
+model = pickle.load(open('resources/model.sav' , 'rb'))
+
+
+# Query Function
+def query(query_text, k=5):
+    enc = model.encode([query_text])
+    D, I = index.search(enc, k)
+
+    results = []
+    for i in range(k):
+        results.append(
+            {
+                "Title" : titles[I[0][i]],
+                "Author" : authors[I[0][i]],
+                "Year" : years[I[0][i]],
+                "Summary" : summary[I[0][i]],
+                "Title URL" : title_urls[I[0][i]]
+            }
+        )
+    
+    return results
+
+
+def main():
+    # Streamlit App
+    st.title("Research Paper/ Publication Search Engine")
+
+    # app description
+    st.markdown(f"<p style='text-align: justify;'>This is a search engine that uses a pre-trained sentence transformer model to encode the user query and the abstracts of the papers. The encoded vectors are then used to find the most similar papers to the user query.", unsafe_allow_html=True)
+
+    # DISCLAIMER: Data capture
+    # st.markdown(f"<p style='text-align: justify;'>Please note that the search engine captures the user query and stores it in a text file. This is done to build a collection of user queries for future use when building a bigger data pool for the search engine.  \nline", unsafe_allow_html=True)
+    
+
+    query_text = st.text_input("Enter your query e.g. NLP")
+
+    if st.button("Search"):
+        results = query(query_text)
+        
+        for result in results:
+            st.write("Title:   " + result["Title"])
+            st.write("Author:   " + result["Author"])
+            st.write("Year Published:   " +  str(result["Year"]))
+            
+            # justified text for summary
+            st.write("Abstract")
+            st.markdown(f"<p style='text-align: justify;'>{result['Summary']}</p>", unsafe_allow_html=True)
+            st.markdown(f"<a href='{result['Title URL']}'>Google Search</a>", unsafe_allow_html=True)
+            st.write("--" * 50)
+        
+
+        # store user query in a text file, 
+        # The purpose of this is to build a collection of user queries for future use when building a bigger data pool for the search engine
+        # with open('resources/user_query.txt', 'a') as f:
+        #    f.write(query_text + "\n")'''
+
+if __name__ == "__main__":
+    main()

requirements.txt

@@ -0,0 +1,3 @@
+faiss-cpu
+numpy
+sentence-transformers