| {"guide": {"name": "creating-a-chatbot-fast", "category": "chatbots", "pretty_category": "Chatbots", "guide_index": 1, "absolute_index": 25, "pretty_name": "Creating A Chatbot Fast", "content": "# How to Create a Chatbot with Gradio\n\n\n\n## Introduction\n\nChatbots are a popular application of large language models. Using `gradio`, you can easily build a demo of your chatbot model and share that with your users, or try it yourself using an intuitive chatbot UI.\n\nThis tutorial uses `gr.ChatInterface()`, which is a high-level abstraction that allows you to create your chatbot UI fast, often with a single line of code. The chatbot interface that we create will look something like this:\n\n<gradio-app space='gradio/chatinterface_streaming_echo'></gradio-app>\n\nWe'll start with a couple of simple examples, and then show how to use `gr.ChatInterface()` with real language models from several popular APIs and libraries, including `langchain`, `openai`, and Hugging Face.\n\n**Prerequisites**: please make sure you are using the **latest version** version of Gradio:\n\n```bash\n$ pip install --upgrade gradio\n```\n\n## Defining a chat function\n\nWhen working with `gr.ChatInterface()`, the first thing you should do is define your chat function. Your chat function should take two arguments: `message` and `history` (the arguments can be named anything, but must be in this order).\n\n- `message`: a `str` representing the user's input.\n- `history`: If you set `type=\"messages\"` in gr.ChatInterface, the history will be a list of dictionaries with `role` and `content` keys. Please see the chatbot [docs](/docs/gradio/chatbot) for an in-depth explanation of the chatbot format. \n\nHere is an example value of the `history`:\n\n```python\n[\n {\"role\": \"user\", \"content\": \"What is the capital of France\"},\n {\"role\": \"assistant\", \"content\": \"Paris\"}\n]\n```\n\n\nYour function should return a single string response, which is the bot's response to the particular user input `message`. Your function can take into account the `history` of messages, as well as the current message.\n <p class='tip'>\n <span class=\"inline-flex\" style=\"align-items: baseline\">\n <svg class=\"self-center w-5 h-5 mx-1\" xmlns=\"http://www.w3.org/2000/svg\" width=\"800px\" height=\"800px\" viewBox=\"0 0 24 24\" fill=\"currentColor\">\n <path fill-rule=\"evenodd\" clip-rule=\"evenodd\" d=\"M9.25 18.7089C9.25 18.2894 9.58579 17.9494 10 17.9494H14C14.4142 17.9494 14.75 18.2894 14.75 18.7089C14.75 19.1283 14.4142 19.4684 14 19.4684H10C9.58579 19.4684 9.25 19.1283 9.25 18.7089ZM9.91667 21.2405C9.91667 20.821 10.2525 20.481 10.6667 20.481H13.3333C13.7475 20.481 14.0833 20.821 14.0833 21.2405C14.0833 21.66 13.7475 22 13.3333 22H10.6667C10.2525 22 9.91667 21.66 9.91667 21.2405Z\"/>\n <path d=\"M7.41058 13.8283L8.51463 14.8807C8.82437 15.1759 9 15.5875 9 16.0182C9 16.6653 9.518 17.1899 10.157 17.1899H13.843C14.482 17.1899 15 16.6653 15 16.0182C15 15.5875 15.1756 15.1759 15.4854 14.8807L16.5894 13.8283C18.1306 12.3481 18.9912 10.4034 18.9999 8.3817L19 8.29678C19 4.84243 15.866 2 12 2C8.13401 2 5 4.84243 5 8.29678L5.00007 8.3817C5.00875 10.4034 5.86939 12.3481 7.41058 13.8283Z\"/>\n </svg>\n <span><strong>Tip:</strong></span>\n </span>\n It's strongly recommended to set type=\"messages\" in gr.ChatInterface. Setting type=\"tuples\" is deprecated and will be removed in a future version of Gradio.\n </p>\n \n\nLet's take a look at a few example applications.\n\n## Example: a chatbot that responds yes or no\n\nLet's write a chat function that responds `Yes` or `No` randomly.\n\nHere's our chat function:\n\n```python\nimport random\n\ndef random_response(message, history):\n return random.choice([\"Yes\", \"No\"])\n```\n\nNow, we can plug this into `gr.ChatInterface()` and call the `.launch()` method to create the web interface:\n\n```python\nimport gradio as gr\n\ngr.ChatInterface(random_response, type=\"messages\").launch()\n```\n\nThat's it! Here's our running demo, try it out:\n\n<gradio-app space='gradio/chatinterface_random_response'></gradio-app>\n\n## Another example using the user's input and history\n\nOf course, the previous example was very simplistic, it didn't even take user input or the previous history into account! Here's another simple example showing how to incorporate a user's input as well as the history.\n\n```python\nimport random\nimport gradio as gr\n\ndef alternatingly_agree(message, history):\n if len([h for h in history if h['role'] == \"assistant\"]) % 2 == 0:\n return f\"Yes, I do think that '{message}'\"\n else:\n return \"I don't think so\"\n\ngr.ChatInterface(alternatingly_agree, type=\"messages\").launch()\n```\n\n## Streaming chatbots\n\nIn your chat function, you can use `yield` to generate a sequence of partial responses, each replacing the previous ones. This way, you'll end up with a streaming chatbot. It's that simple!\n\n```python\nimport time\nimport gradio as gr\n\ndef slow_echo(message, history):\n for i in range(len(message)):\n time.sleep(0.3)\n yield \"You typed: \" + message[: i+1]\n\ngr.ChatInterface(slow_echo, type=\"messages\").launch()\n```\n\n <p class='tip'>\n <span class=\"inline-flex\" style=\"align-items: baseline\">\n <svg class=\"self-center w-5 h-5 mx-1\" xmlns=\"http://www.w3.org/2000/svg\" width=\"800px\" height=\"800px\" viewBox=\"0 0 24 24\" fill=\"currentColor\">\n <path fill-rule=\"evenodd\" clip-rule=\"evenodd\" d=\"M9.25 18.7089C9.25 18.2894 9.58579 17.9494 10 17.9494H14C14.4142 17.9494 14.75 18.2894 14.75 18.7089C14.75 19.1283 14.4142 19.4684 14 19.4684H10C9.58579 19.4684 9.25 19.1283 9.25 18.7089ZM9.91667 21.2405C9.91667 20.821 10.2525 20.481 10.6667 20.481H13.3333C13.7475 20.481 14.0833 20.821 14.0833 21.2405C14.0833 21.66 13.7475 22 13.3333 22H10.6667C10.2525 22 9.91667 21.66 9.91667 21.2405Z\"/>\n <path d=\"M7.41058 13.8283L8.51463 14.8807C8.82437 15.1759 9 15.5875 9 16.0182C9 16.6653 9.518 17.1899 10.157 17.1899H13.843C14.482 17.1899 15 16.6653 15 16.0182C15 15.5875 15.1756 15.1759 15.4854 14.8807L16.5894 13.8283C18.1306 12.3481 18.9912 10.4034 18.9999 8.3817L19 8.29678C19 4.84243 15.866 2 12 2C8.13401 2 5 4.84243 5 8.29678L5.00007 8.3817C5.00875 10.4034 5.86939 12.3481 7.41058 13.8283Z\"/>\n </svg>\n <span><strong>Tip:</strong></span>\n </span>\n While the response is streaming, the \"Submit\" button turns into a \"Stop\" button that can be used to stop the generator function. You can customize the appearance and behavior of the \"Stop\" button using the `stop_btn` parameter.\n </p>\n \n\n## Customizing your chatbot\n\nIf you're familiar with Gradio's `Interface` class, the `gr.ChatInterface` includes many of the same arguments that you can use to customize the look and feel of your Chatbot. For example, you can:\n\n- add a title and description above your chatbot using `title` and `description` arguments.\n- add a theme or custom css using `theme` and `css` arguments respectively.\n- add `examples` and even enable `cache_examples`, which make it easier for users to try it out. `examples` can be customized by adding `display_icon` or `display_text` keys to each example.\n- You can change the text or disable each of the buttons that appear in the chatbot interface: `submit_btn`, `retry_btn`, `undo_btn`, `clear_btn`.\n\nIf you want to customize the `gr.Chatbot` or `gr.Textbox` that compose the `ChatInterface`, then you can pass in your own chatbot or textbox as well. Here's an example of how we can use these parameters:\n\n```python\nimport gradio as gr\n\ndef yes_man(message, history):\n if message.endswith(\"?\"):\n return \"Yes\"\n else:\n return \"Ask me anything!\"\n\ngr.ChatInterface(\n yes_man,\n type=\"messages\",\n chatbot=gr.Chatbot(height=300),\n textbox=gr.Textbox(placeholder=\"Ask me a yes or no question\", container=False, scale=7),\n title=\"Yes Man\",\n description=\"Ask Yes Man any question\",\n theme=\"soft\",\n examples=[{\"text\": \"Hello\"}, {\"text\": \"Am I cool?\"}, {\"text\": \"Are tomatoes vegetables?\"}],\n cache_examples=True,\n retry_btn=None,\n undo_btn=\"Delete Previous\",\n clear_btn=\"Clear\",\n).launch()\n```\n\nIn particular, if you'd like to add a \"placeholder\" for your chat interface, which appears before the user has started chatting, you can do so using the `placeholder` argument of `gr.Chatbot`, which accepts Markdown or HTML. \n\n```python\ngr.ChatInterface(\n yes_man,\n type=\"messages\",\n chatbot=gr.Chatbot(placeholder=\"<strong>Your Personal Yes-Man</strong><br>Ask Me Anything\"),\n...\n```\n\nThe placeholder appears vertically and horizontally centered in the chatbot.\n\n## Add Multimodal Capability to your chatbot\n\nYou may want to add multimodal capability to your chatbot. For example, you may want users to be able to easily upload images or files to your chatbot and ask questions about it. You can make your chatbot \"multimodal\" by passing in a single parameter (`multimodal=True`) to the `gr.ChatInterface` class.\n\n\n```python\nimport gradio as gr\n\ndef count_files(message, history):\n num_files = len(message[\"files\"])\n return f\"You uploaded {num_files} files\"\n\ndemo = gr.ChatInterface(fn=count_files, type=\"messages\", examples=[{\"text\": \"Hello\", \"files\": []}], title=\"Echo Bot\", multimodal=True)\n\ndemo.launch()\n```\n\nWhen `multimodal=True`, the signature of `fn` changes slightly. The first parameter of your function should accept a dictionary consisting of the submitted text and uploaded files that looks like this: `{\"text\": \"user input\", \"file\": [\"file_path1\", \"file_path2\", ...]}`. Similarly, any examples you provide should be in a dictionary of this form. Your function should still return a single `str` message. \n <p class='tip'>\n <span class=\"inline-flex\" style=\"align-items: baseline\">\n <svg class=\"self-center w-5 h-5 mx-1\" xmlns=\"http://www.w3.org/2000/svg\" width=\"800px\" height=\"800px\" viewBox=\"0 0 24 24\" fill=\"currentColor\">\n <path fill-rule=\"evenodd\" clip-rule=\"evenodd\" d=\"M9.25 18.7089C9.25 18.2894 9.58579 17.9494 10 17.9494H14C14.4142 17.9494 14.75 18.2894 14.75 18.7089C14.75 19.1283 14.4142 19.4684 14 19.4684H10C9.58579 19.4684 9.25 19.1283 9.25 18.7089ZM9.91667 21.2405C9.91667 20.821 10.2525 20.481 10.6667 20.481H13.3333C13.7475 20.481 14.0833 20.821 14.0833 21.2405C14.0833 21.66 13.7475 22 13.3333 22H10.6667C10.2525 22 9.91667 21.66 9.91667 21.2405Z\"/>\n <path d=\"M7.41058 13.8283L8.51463 14.8807C8.82437 15.1759 9 15.5875 9 16.0182C9 16.6653 9.518 17.1899 10.157 17.1899H13.843C14.482 17.1899 15 16.6653 15 16.0182C15 15.5875 15.1756 15.1759 15.4854 14.8807L16.5894 13.8283C18.1306 12.3481 18.9912 10.4034 18.9999 8.3817L19 8.29678C19 4.84243 15.866 2 12 2C8.13401 2 5 4.84243 5 8.29678L5.00007 8.3817C5.00875 10.4034 5.86939 12.3481 7.41058 13.8283Z\"/>\n </svg>\n <span><strong>Tip:</strong></span>\n </span>\n If you'd like to customize the UI/UX of the textbox for your multimodal chatbot, you should pass in an instance of `gr.MultimodalTextbox` to the `textbox` argument of `ChatInterface` instead of an instance of `gr.Textbox`.\n </p>\n \n\n## Additional Inputs\n\nYou may want to add additional parameters to your chatbot and expose them to your users through the Chatbot UI. For example, suppose you want to add a textbox for a system prompt, or a slider that sets the number of tokens in the chatbot's response. The `ChatInterface` class supports an `additional_inputs` parameter which can be used to add additional input components.\n\nThe `additional_inputs` parameters accepts a component or a list of components. You can pass the component instances directly, or use their string shortcuts (e.g. `\"textbox\"` instead of `gr.Textbox()`). If you pass in component instances, and they have _not_ already been rendered, then the components will appear underneath the chatbot within a `gr.Accordion()`. You can set the label of this accordion using the `additional_inputs_accordion_name` parameter.\n\nHere's a complete example:\n\n```python\nimport gradio as gr\nimport time\n\ndef echo(message, history, system_prompt, tokens):\n response = f\"System prompt: {system_prompt}\\n Message: {message}.\"\n for i in range(min(len(response), int(tokens))):\n time.sleep(0.05)\n yield response[: i + 1]\n\ndemo = gr.ChatInterface(\n echo,\n type=\"messages\",\n additional_inputs=[\n gr.Textbox(\"You are helpful AI.\", label=\"System Prompt\"),\n gr.Slider(10, 100),\n ],\n)\n\ndemo.launch()\n\n```\n\nIf the components you pass into the `additional_inputs` have already been rendered in a parent `gr.Blocks()`, then they will _not_ be re-rendered in the accordion. This provides flexibility in deciding where to lay out the input components. In the example below, we position the `gr.Textbox()` on top of the Chatbot UI, while keeping the slider underneath.\n\n```python\nimport gradio as gr\nimport time\n\ndef echo(message, history, system_prompt, tokens):\n response = f\"System prompt: {system_prompt}\\n Message: {message}.\"\n for i in range(min(len(response), int(tokens))):\n time.sleep(0.05)\n yield response[: i+1]\n\nwith gr.Blocks() as demo:\n system_prompt = gr.Textbox(\"You are helpful AI.\", label=\"System Prompt\")\n slider = gr.Slider(10, 100, render=False)\n\n gr.ChatInterface(\n echo, additional_inputs=[system_prompt, slider], type=\"messages\"\n )\n\ndemo.launch()\n```\n\nIf you need to create something even more custom, then its best to construct the chatbot UI using the low-level `gr.Blocks()` API. We have [a dedicated guide for that here](/guides/creating-a-custom-chatbot-with-blocks).\n\n## Displaying Files or Components in the Chatbot\n\nThe `Chatbot` component supports using many of the core Gradio components (such as `gr.Image`, `gr.Plot`, `gr.Audio`, and `gr.HTML`) inside of the chatbot. Simply return one of these components from your function to use it with `gr.ChatInterface`. Here's an example:\n\n```py\nimport gradio as gr\n\ndef fake(message, history):\n if message.strip():\n return gr.Audio(\"https://github.com/gradio-app/gradio/raw/main/test/test_files/audio_sample.wav\")\n else:\n return \"Please provide the name of an artist\"\n\ngr.ChatInterface(\n fake,\n type=\"messages\",\n textbox=gr.Textbox(placeholder=\"Which artist's music do you want to listen to?\", scale=7),\n chatbot=gr.Chatbot(placeholder=\"Play music by any artist!\"),\n).launch()\n```\n\nYou can also return a dictionary with a `path` key that points to a local file or a publicly available URL.\n\n```py\nimport gradio as gr\n\ndef fake(message, history):\n if message.strip():\n return {\"role\": \"assistant\", \"content\": {\"path\": \"https://github.com/gradio-app/gradio/raw/main/test/test_files/audio_sample.wav\"}}\n else:\n return \"Please provide the name of an artist\"\n\ngr.ChatInterface(\n fake,\n type=\"messages\",\n textbox=gr.Textbox(placeholder=\"Which artist's music do you want to listen to?\", scale=7),\n chatbot=gr.Chatbot(placeholder=\"Play music by any artist!\"),\n).launch()\n```\n\n See the chatbot [docs](/docs/gradio/chatbot#behavior) for an explanation how.\n\n## Using your chatbot via an API\n\nOnce you've built your Gradio chatbot and are hosting it on [Hugging Face Spaces](https://hf.space) or somewhere else, then you can query it with a simple API at the `/chat` endpoint. The endpoint just expects the user's message (and potentially additional inputs if you have set any using the `additional_inputs` parameter), and will return the response, internally keeping track of the messages sent so far.\n\n[](https://github.com/gradio-app/gradio/assets/1778297/7b10d6db-6476-4e2e-bebd-ecda802c3b8f)\n\nTo use the endpoint, you should use either the [Gradio Python Client](/guides/getting-started-with-the-python-client) or the [Gradio JS client](/guides/getting-started-with-the-js-client).\n\n## A `langchain` example\n\nNow, let's actually use the `gr.ChatInterface` with some real large language models. We'll start by using `langchain` on top of `openai` to build a general-purpose streaming chatbot application in 19 lines of code. You'll need to have an OpenAI key for this example (keep reading for the free, open-source equivalent!)\n\n```python\nfrom langchain.chat_models import ChatOpenAI\nfrom langchain.schema import AIMessage, HumanMessage\nimport openai\nimport gradio as gr\n\nos.environ[\"OPENAI_API_KEY\"] = \"sk-...\" # Replace with your key\n\nllm = ChatOpenAI(temperature=1.0, model='gpt-3.5-turbo-0613')\n\ndef predict(message, history):\n history_langchain_format = []\n for msg in history:\n if msg['role'] == \"user\":\n history_langchain_format.append(HumanMessage(content=msg['content']))\n elif msg['role'] == \"assistant\":\n history_langchain_format.append(AIMessage(content=msg['content']))\n history_langchain_format.append(HumanMessage(content=message))\n gpt_response = llm(history_langchain_format)\n return gpt_response.content\n\ngr.ChatInterface(predict, type=\"messages\").launch()\n```\n\n## A streaming example using `openai`\n\nOf course, we could also use the `openai` library directy. Here a similar example, but this time with streaming results as well:\n\n```python\nfrom openai import OpenAI\nimport gradio as gr\n\napi_key = \"sk-...\" # Replace with your key\nclient = OpenAI(api_key=api_key)\n\ndef predict(message, history):\n history_openai_format = []\n for msg in history:\n history_openai_format.append(msg)\n history_openai_format.append(message)\n \n response = client.chat.completions.create(model='gpt-3.5-turbo',\n messages= history_openai_format,\n temperature=1.0,\n stream=True)\n\n partial_message = \"\"\n for chunk in response:\n if chunk.choices[0].delta.content is not None:\n partial_message = partial_message + chunk.choices[0].delta.content\n yield partial_message\n\ngr.ChatInterface(predict, type=\"messages\").launch()\n```\n\n**Handling Concurrent Users with Threads**\n\nThe example above works if you have a single user \u2014 or if you have multiple users, since it passes the entire history of the conversation each time there is a new message from a user. \n\nHowever, the `openai` library also provides higher-level abstractions that manage conversation history for you, e.g. the [Threads abstraction](https://platform.openai.com/docs/assistants/how-it-works/managing-threads-and-messages). If you use these abstractions, you will need to create a separate thread for each user session. Here's a partial example of how you can do that, by accessing the `session_hash` within your `predict()` function:\n\n```py\nimport openai\nimport gradio as gr\n\nclient = openai.OpenAI(api_key = os.getenv(\"OPENAI_API_KEY\"))\nthreads = {}\n\ndef predict(message, history, request: gr.Request):\n if request.session_hash in threads:\n thread = threads[request.session_hash]\n else:\n threads[request.session_hash] = client.beta.threads.create()\n \n message = client.beta.threads.messages.create(\n thread_id=thread.id,\n role=\"user\",\n content=message)\n \n ...\n\ngr.ChatInterface(predict, type=\"messages\").launch()\n```\n\n## Example using a local, open-source LLM with Hugging Face\n\nOf course, in many cases you want to run a chatbot locally. Here's the equivalent example using Together's RedePajama model, from Hugging Face (this requires you to have a GPU with CUDA).\n\n```python\nimport gradio as gr\nimport torch\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, StoppingCriteria, StoppingCriteriaList, TextIteratorStreamer\nfrom threading import Thread\n\ntokenizer = AutoTokenizer.from_pretrained(\"togethercomputer/RedPajama-INCITE-Chat-3B-v1\")\nmodel = AutoModelForCausalLM.from_pretrained(\"togethercomputer/RedPajama-INCITE-Chat-3B-v1\", torch_dtype=torch.float16)\nmodel = model.to('cuda:0')\n\nclass StopOnTokens(StoppingCriteria):\n def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor, **kwargs) -> bool:\n stop_ids = [29, 0]\n for stop_id in stop_ids:\n if input_ids[0][-1] == stop_id:\n return True\n return False\n\ndef predict(message, history):\n history_transformer_format = list(zip(history[:-1], history[1:])) + [[message, \"\"]]\n stop = StopOnTokens()\n\n messages = \"\".join([\"\".join([\"\\n<human>:\"+item[0], \"\\n<bot>:\"+item[1]])\n for item in history_transformer_format])\n\n model_inputs = tokenizer([messages], return_tensors=\"pt\").to(\"cuda\")\n streamer = TextIteratorStreamer(tokenizer, timeout=10., skip_prompt=True, skip_special_tokens=True)\n generate_kwargs = dict(\n model_inputs,\n streamer=streamer,\n max_new_tokens=1024,\n do_sample=True,\n top_p=0.95,\n top_k=1000,\n temperature=1.0,\n num_beams=1,\n stopping_criteria=StoppingCriteriaList([stop])\n )\n t = Thread(target=model.generate, kwargs=generate_kwargs)\n t.start()\n\n partial_message = \"\"\n for new_token in streamer:\n if new_token != '<':\n partial_message += new_token\n yield partial_message\n\ngr.ChatInterface(predict).launch()\n```\n\nWith those examples, you should be all set to create your own Gradio Chatbot demos soon! For building even more custom Chatbot applications, check out [a dedicated guide](/guides/creating-a-custom-chatbot-with-blocks) using the low-level `gr.Blocks()` API.\n", "tags": ["NLP", "TEXT", "CHAT"], "spaces": [], "url": "/guides/creating-a-chatbot-fast/", "contributor": null}} |