Spaces:
Paused
Paused
Working of Search Web
#339
by
Saugatkafley
- opened
Hello , I wanted to learn about the Search web feature?
- does this relate with some advanced form of RAG?
- How is the most recent information is captured?
- Which sites does it looks upon?
Hi! You can look at the entrypoint for the websearch on github here. To summarize:
- We generate a google query based on the conversation history
- That query is passed to a search results provider (we support 4 options, but in production we use serper.dev)
- We fetch the content of the top webpages returned by our search result provider
- We extract the text from those webpages, chunk it up and use transformers.js to generate embeddings for them
- We fetch the most relevant chunks using similarity search
- We inject the most relevant chunks in the conversation context before generating the answer.
Hope that answers your question, again all the code is available on github so feel free to have a look there for yourself if you're missing something.