[NEW] Internet access for Assistants

#385
by nsarrazin HF staff - opened
Hugging Chat org
โ€ข
edited Mar 25

image (2).png

Hey! We have just released an update to HuggingChat Assistants that allows you to connect them to the Internet to get more relevant and interactive answers. When you create or edit an Assistant, you will now see an option for Internet access. It can have four settings: enabled, domain search, specific links, and disabled.

  • Enabled is the same websearch we currently use in HuggingChat. Use it for generic assistants that can have conversations about many domains.
  • Domains search allows you to specify a domain name or even part of a website that the web search will crawl to try to find relevant information. Useful if you want to have an Assistant that can find search and use content from your website or a particular news outlet you like, for example.
  • Specific Links lets you specify a direct URL to a web page or plain text document that you want to pass to the Assistant. This is very useful for talking to your text documents, adding extra context to your role-playing game, or passing any arbitrary data from your web server to the Assistant.
  • Dynamic links Allow the use of template variables {{url=https://example.com/path}} to insert dynamic content into your prompt by making GET requests to specified URLs on each inference.

Enabled

This is equivalent to enabling the web search toggle in HuggingChat. The model generates a query from your question, uses it to search the web, and parses the results to improve the answer. You can't restrict which domains or links are used in this mode.

Domains search

In this mode you can restrict the domains used in the web search. This is equivalent to appending site:example.com to the web query. For example if you want to make an assistant that only uses wikipedia as a source:

You can also be more specific, for example you could have an assistant that only knows about the docs for our library diffusers.

Specific Links

This allows you to directly specify up to 10 URLs that will be added directly to the context for better results. We support both HTML and plain text content such as markdown! For example, if you want your assistant to know about the chat-ui repository README file, you can add a link to the markdown file:

Or if you want an assistant that knows about the current top news headlines:

This is a very flexible setting because you have complete control over what is passed to the context. For example, you could create your own web server in a space that returns a different country name every day and use it to make a country guessing game where every user gets the same new country every day. Also note that for long web pages it will limit the maximum context it can use, so less may perform better.

Safety & Trust

Because web-connected assistants use an external source of information, you should never trust any information without verifying it yourself, and you should always manually check which source they use; this information is publicly available on the Assistant Settings page. Web-connected assistants have a special icon so you can easily identify them.

Let us know what you think of the feature, and feel free to share your new assistants in this thread! ๐Ÿค—

It doesn't look like it's been pushed to huggingface.co/chat, is this a bug?

Hugging Chat org

@EveryPizza Feel free to check again! Should be there now

Hugging Chat org

this is incredible ๐Ÿ”ฅ

Love itt! Great job team!

Can the specific domains option be used in some clever way with a cyoa game, where the user mostly inputs numbers?
I tried it here, but web fails: https://hf.co/chat/assistant/65c80b2288d76ebc0fc357ca

victor pinned discussion

I m afraid that the Domains search option is not working correctly

I have just created an assistant for the records in the President John F. Kennedy Assassination Records Collection

https://hf.co/chat/assistant/65f4604731b538d22b432df7

I need it to answer based only in material found on archives.gov subdomains but, alas, it returns text from reddit, wikipedia and other sources

Hugging Chat org
โ€ข
edited Mar 15

@emilios can you share a conversation where this doesn't work for you? because for me it works: https://hf.co/chat/r/DN9CIVy

Can you check this please?

https://hf.co/chat/r/lWWJp8E

Hugging Chat org

Ok interesting seems like there's indeed an issue, let me have a look!

Well the good news is that it did not happen again the last 5 hours :)

Hugging Chat org

I pushed a fix for it today, shouldn't happen again! @emilios

Is there a way in the system prompt or otherwise to have the web search not be used for every response? im finding that for my testing of a coding helper that if it gives its initial response and used web search and my reply is continue or anything really for it to continue with the process it outlined it searches the web for continue instead of continuing

  • note i have gotten it to not have this issue and keep responding in the context of the project implementation but it still searches web every time which would be nice to toggle
Hugging Chat org

Good point @Csplk we'll see what we can do.

Good point @Csplk we'll see what we can do.

Thanks

Awesome, it just works perfectly !
I have just automated my assistant with live data.
I'll make a video probably today to celebrate this.

This comment has been hidden

It works perfectly well:

There seems to be an occasional problem, like a content mismatch, when using
Dynamic Prompt

Sometimes the results are allright, but some other times they are confusing
I m guessing it either hallucinates or sometimes it cannot access the web

I've tried {{url=https://www.imdb.com/title/tt14539740/}} Godzilla x Kong
but I got https://www.imdb.com/title/tt13320622/ Lost City

https://hf.co/chat/r/giJJUN8

And it is not related to the LLM you choose

Hugging Chat org
โ€ข
edited 26 days ago

@emilios this might not be super clear from the current layout but dynamic prompt only injects web pages in the assistant system instructions. You would have to put your URL in your system instructions for them to be picked up.

Thank you for your answer. Since we can give a specific url with other ways in the instructions for dynamic content, would n't it be nice if something like {{url=}} is used in the chat prompt to parse the text of a different url every time needed?

@nsarrazin
image.png

I think you are using google json api and that's the reason it stops working.

Hugging Chat org

Sry @KingNish it's back working!

This comment has been hidden

Hi all, and thank you HF team for making this awesome feature available. After much experimentation and failed attempts, it seems I finally managed to make an assistant that works for finding relevant content on my website with the "domains search" option. For anyone is interested in the same use case, you can check my system prompt and conversation starters. I have observed that when asking not to generate URLs as part of the response (the links are generated by the app) and sending very simple prompts ("Show me content about...") it does what is expected most of the time and provides links to relevant pages and articles along with a coherent text intro:

Talking to Chatbots Web Browser (assistant)

For general web browsing, here's another assistant I built and works fine so far: Web Browsing Chatbot (assistant)

A potential improvement for this feature would be to connect asisstant with APIs, so content retrieval coul be refined (for example, leveraging Wordpress search on blogs built in WP) or incorporate any sort of interaction with a website that can be defined with an API.

I have switched on Command R and it also rocks !!!

I have switched on Command R and it also rocks !!!

Same, It's even better than Mixtrail, (Mixtrail sometimes repeat same thing or sometime system prompt, I told him to do not do this but it does not stope permanently, but when i switched to Comman R+ it works.)

I have switched on Command R and it also rocks !!!

Same, It's even better than Mixtrail, (Mixtrail sometimes repeat same thing or sometime system prompt, I told him to do not do this but it does not stope permanently, but when i switched to Comman R+ it works.)

I was on Nous ;-p

I was on Nous ;-p

Nous was better than mixtrail in creative work and Mixtrail is better in specific text generation work.
so, it depends on your work type.

But it also have some wrong information
image.png

like he was made by open ai, and many more.

it is possible to pass argument to "Specific links" as the query from the user ?

it is possible to pass argument to "Specific links" as the query from the user ?

Yes, use dynamic links.

it is possible to pass argument to "Specific links" as the query from the user ?

Yes, use dynamic links.

but what is the syntax in order to send user input to the to dunamic links? {{prompt}} does not work

Hugging Chat org

but what is the syntax in order to send user input to the to dunamic links? {{prompt}} does not work

We are currently working on something to do this ๐Ÿ‘€

Could you add a button to enable or disable internet search if the chatbot allows it, as in your default models? This would prevent the chatbot from automatically using internet search every time a prompt is sent.

In the case the resource is an url targetting an image, would it possible to display it directly in the chat ? Currently I can get the url but I have to click on it to see the image in another tab.

Could you add a button to enable or disable internet search if the chatbot allows it, as in your default models? This would prevent the chatbot from automatically using internet search every time a prompt is sent.

Just conclude in Systm prompt when to use web search and when not

In the case the resource is an url targetting an image, would it possible to display it directly in the chat ? Currently I can get the url but I have to click on it to see the image in another tab.

Use url in format

![](url)

example - ![](https://source.unsplash.com/random/?photo)

but what is the syntax in order to send user input to the dynamic links? {{prompt}} does not work.

Conclude in prompt that {prompt} is prompt given by user.

Here is an example - https://hf.co/chat/r/-XIrb_g

System prompt: -
Answer the query given by user from this site:
https://en.wikipedia.org/wiki/{prompt}
Here {prompt} is prompt given by user.

@KingNish : I'm going to try to tune the SYSTEM prompt to ask... when an image url is found, to answer so

@KingNish : I'm going to try to tune the SYSTEM prompt to ask... when an image url is found, to answer so

Can you give link to your bot

@rastadidi Are you trying to make bot which can identify objects in image or answer the question asked by user.

@KingNish :

@rastadidi Are you trying to make bot which can identify objects in image or answer the question asked by user.

Indeed I would lke him to enhance the answer with images
Chatbot : https://hf.co/chat/assistant/65bd4d47a16aaa191b5b501d

(the image url is part of a csv)

@rastadidi see this https://hf.co/chat/r/OjN_g8M

Image url are not working before because of this
image.png

i shifted it to hugging face database and it starts working

but what is the syntax in order to send user input to the to dynamic links? {{prompt}} does not work

Now Dynamic links are working.

@KingNish , you mentioned that we should 'Just conclude in System prompt when to use web search and when not,' as you can see in your response here: https://huggingface.co/spaces/huggingchat/chat-ui/discussions/385#661e39348d2ef5cea2837fd1. I was suggesting adding this button to our customized chatbot because if we enable this option (Web Research), all responses will come solely from web research. Unlike the main HuggingChat, where we have this button as an option, we can activate it whenever we choose to.

@IRZOUNI OOo, something like this
image.png
+1 for this feature

@KingNish Definitely yes!!! that would be very much appreciated. Thank you.

Can you open the prompt customization of the Internet search model? I have been trying to rewrite papers with cR+ recently, but it can only make up fake references, and it often gets stuck after enabling Internet search.

Any update on dynamic links where API URL with arguments is provided?

Anyone have problems with the web search not working? it just says generation failed
Screenshot 2024-04-19 at 7.17.32โ€ฏPM.png

I think that there should also be the option to enable or disable the web search for each specific query within the chat just like in normal (non-assistant) chats.

Hugging Chat org

I think that there should also be the option to enable or disable the web search for each specific query within the chat just like in normal (non-assistant) chats.

Or integrate it as a tool that can be called by the model (or not) by himself? we are working on something like this.

I think that there should also be the option to enable or disable the web search for each specific query within the chat just like in normal (non-assistant) chats.

Or integrate it as a tool that can be called by the model (or not) by himself? we are working on something like this.

I think they can both be helpful. Personally I like the toggle option because I have more control over it. But I also get that the tool use might make the experience a little more fluent (if it works well enough).

Or integrate it as a tool that can be called by the model (or not) by himself? we are working on something like this.

This is the only correct solution. The existing search is essentially to let other models guess what we want to search for, which is far less accurate than letting the model master function call by itself. And function calls can easily add mathematics and other abilities to the model, with coding cost-effective.

You mean... we could build then embed third party custom tools ?... "ร  la langchain" ? That will be awesome !

Assistant should be able to converse and interact with the user. Only doing an internet search when it is useful. If I greet the assistant, for example, I do not want my greeting to trigger an internet search. Essentially, the assistant should use the internet for information retrieval, and they should know when it is necessary for an internet search. ๐Ÿ˜

don't tell anyone but this is coming @Joseph717171 (btw sometimes I want to decide to use websearch or not by myself vs always letting the model decide)

It would be great to add a feature to custom assistants, which have been given internet capabilities, to disable/enable them with a button, as is the case with the models on the page https://huggingface.co/chat/models.

I also think it would be good to disable the option that hides/unhides the copy to clipboard, retry buttons. On a PC, it is sometimes difficult to get these buttons to appear and to click on them to copy to the clipboard.

Thanks for everything you do, the Hugging Face team, its approach, its free open source offering. I love you guys.

Sign up or log in to comment