docs/sorbobot.md

Sorbobot: Expert Finder Chatbot Documentation

Overview

Sorbobot is a chatbot designed for Sorbonne Université to assist their administration in locating academic experts within the university. This document outlines the structure, functionality, and implementation details of Sorbobot.

Context

Sorbobot centers around identifying experts with precision, avoiding confusion with individuals sharing similar names. It leverages HAL unique identifiers to distinguish between experts.

System Architecture

Sorbobot operates on a Retrieval Augmented Generation (RAG) system, composed of two primary steps:

1. Retrieval: Identifies publications most similar to the user queries.
2. Generation: Produces responses based on the context extracted from relevant publications.

Implementation Details

Programming Language and Libraries

• Language: Python
• Frontend: Streamlit
• Database: PostgreSQL with pgvector for similarity search
• NLP Processing: langchain and GPT4all libraries

Database

• Postgres with pgvector: Used for storing data and performing similarity searches based on cosine similarity metrics.

Natural Language Processing

• Abstracts as Data Source: The chatbot utilizes publication abstracts to identify experts.
• GPT4all for Word Embedding: Converts text from author publications into word embeddings, enhancing the accuracy of expert identification.

Retrieval Process

1. Query Processing: User queries are processed to extract key terms.
2. Similarity Search: The system searches the database using pgvector to find publications with low cosine distance to the query.
3. Expert Identification: The system identifies authors of these publications, ensuring unique identification of experts.

Generation Process

1. Context Extraction: Relevant information is extracted from the identified publications.
2. Response Generation: Utilizes a LLM to generate informative responses based on the extracted context.

User Interaction Flow

1. Query Submission: Users submit queries related to their expert search.
2. Chatbot Processing: Sorbobot processes the query, retrieves relevant publications, and identifies experts.
3. Response Presentation: The system presents a list of experts, including unique identifiers and relevant publication abstracts.

Conclusion

Sorbobot is a powerful tool for Sorbonne Université, streamlining the process of finding academic experts. Its advanced NLP capabilities, combined with a robust database and intelligent retrieval-generation framework, ensure accurate and efficient expert identification.