{"guide": {"name": "gradio-and-llm-agents", "category": "gradio-clients-and-lite", "pretty_category": "Gradio Clients And Lite", "guide_index": 4, "absolute_index": 51, "pretty_name": "Gradio And Llm Agents", "content": "# Gradio & LLM Agents \ud83e\udd1d\n\nLarge Language Models (LLMs) are very impressive but they can be made even more powerful if we could give them skills to accomplish specialized tasks.\n\nThe [gradio_tools](https://github.com/freddyaboulton/gradio-tools) library can turn any [Gradio](https://github.com/gradio-app/gradio) application into a [tool](https://python.langchain.com/en/latest/modules/agents/tools.html) that an [agent](https://docs.langchain.com/docs/components/agents/agent) can use to complete its task. For example, an LLM could use a Gradio tool to transcribe a voice recording it finds online and then summarize it for you. Or it could use a different Gradio tool to apply OCR to a document on your Google Drive and then answer questions about it.\n\nThis guide will show how you can use `gradio_tools` to grant your LLM Agent access to the cutting edge Gradio applications hosted in the world. Although `gradio_tools` are compatible with more than one agent framework, we will focus on [Langchain Agents](https://docs.langchain.com/docs/components/agents/) in this guide.\n\n## Some background\n\n### What are agents?\n\nA [LangChain agent](https://docs.langchain.com/docs/components/agents/agent) is a Large Language Model (LLM) that takes user input and reports an output based on using one of many tools at its disposal.\n\n### What is Gradio?\n\n[Gradio](https://github.com/gradio-app/gradio) is the defacto standard framework for building Machine Learning Web Applications and sharing them with the world - all with just python! \ud83d\udc0d\n\n## gradio_tools - An end-to-end example\n\nTo get started with `gradio_tools`, all you need to do is import and initialize your tools and pass them to the langchain agent!\n\nIn the following example, we import the `StableDiffusionPromptGeneratorTool` to create a good prompt for stable diffusion, the\n`StableDiffusionTool` to create an image with our improved prompt, the `ImageCaptioningTool` to caption the generated image, and\nthe `TextToVideoTool` to create a video from a prompt.\n\nWe then tell our agent to create an image of a dog riding a skateboard, but to please improve our prompt ahead of time. We also ask\nit to caption the generated image and create a video for it. The agent can decide which tool to use without us explicitly telling it.\n\n```python\nimport os\n\nif not os.getenv(\"OPENAI_API_KEY\"):\n    raise ValueError(\"OPENAI_API_KEY must be set\")\n\nfrom langchain.agents import initialize_agent\nfrom langchain.llms import OpenAI\nfrom gradio_tools import (StableDiffusionTool, ImageCaptioningTool, StableDiffusionPromptGeneratorTool,\n                          TextToVideoTool)\n\nfrom langchain.memory import ConversationBufferMemory\n\nllm = OpenAI(temperature=0)\nmemory = ConversationBufferMemory(memory_key=\"chat_history\")\ntools = [StableDiffusionTool().langchain, ImageCaptioningTool().langchain,\n         StableDiffusionPromptGeneratorTool().langchain, TextToVideoTool().langchain]\n\n\nagent = initialize_agent(tools, llm, memory=memory, agent=\"conversational-react-description\", verbose=True)\noutput = agent.run(input=(\"Please create a photo of a dog riding a skateboard \"\n                          \"but improve my prompt prior to using an image generator.\"\n                          \"Please caption the generated image and create a video for it using the improved prompt.\"))\n```\n\nYou'll note that we are using some pre-built tools that come with `gradio_tools`. Please see this [doc](https://github.com/freddyaboulton/gradio-tools#gradio-tools-gradio--llm-agents) for a complete list of the tools that come with `gradio_tools`.\nIf you would like to use a tool that's not currently in `gradio_tools`, it is very easy to add your own. That's what the next section will cover.\n\n## gradio_tools - creating your own tool\n\nThe core abstraction is the `GradioTool`, which lets you define a new tool for your LLM as long as you implement a standard interface:\n\n```python\nclass GradioTool(BaseTool):\n\n    def __init__(self, name: str, description: str, src: str) -> None:\n\n    @abstractmethod\n    def create_job(self, query: str) -> Job:\n        pass\n\n    @abstractmethod\n    def postprocess(self, output: Tuple[Any] | Any) -> str:\n        pass\n```\n\nThe requirements are:\n\n1. The name for your tool\n2. The description for your tool. This is crucial! Agents decide which tool to use based on their description. Be precise and be sure to include example of what the input and the output of the tool should look like.\n3. The url or space id, e.g. `freddyaboulton/calculator`, of the Gradio application. Based on this value, `gradio_tool` will create a [gradio client](https://github.com/gradio-app/gradio/blob/main/client/python/README.md) instance to query the upstream application via API. Be sure to click the link and learn more about the gradio client library if you are not familiar with it.\n4. create_job - Given a string, this method should parse that string and return a job from the client. Most times, this is as simple as passing the string to the `submit` function of the client. More info on creating jobs [here](https://github.com/gradio-app/gradio/blob/main/client/python/README.md#making-a-prediction)\n5. postprocess - Given the result of the job, convert it to a string the LLM can display to the user.\n6. _Optional_ - Some libraries, e.g. [MiniChain](https://github.com/srush/MiniChain/tree/main), may need some info about the underlying gradio input and output types used by the tool. By default, this will return gr.Textbox() but\n   if you'd like to provide more accurate info, implement the `_block_input(self, gr)` and `_block_output(self, gr)` methods of the tool. The `gr` variable is the gradio module (the result of `import gradio as gr`). It will be\n   automatically imported by the `GradiTool` parent class and passed to the `_block_input` and `_block_output` methods.\n\nAnd that's it!\n\nOnce you have created your tool, open a pull request to the `gradio_tools` repo! We welcome all contributions.\n\n## Example tool - Stable Diffusion\n\nHere is the code for the StableDiffusion tool as an example:\n\n```python\nfrom gradio_tool import GradioTool\nimport os\n\nclass StableDiffusionTool(GradioTool):\n    \"\"\"Tool for calling stable diffusion from llm\"\"\"\n\n    def __init__(\n        self,\n        name=\"StableDiffusion\",\n        description=(\n            \"An image generator. Use this to generate images based on \"\n            \"text input. Input should be a description of what the image should \"\n            \"look like. The output will be a path to an image file.\"\n        ),\n        src=\"gradio-client-demos/stable-diffusion\",\n        hf_token=None,\n    ) -> None:\n        super().__init__(name, description, src, hf_token)\n\n    def create_job(self, query: str) -> Job:\n        return self.client.submit(query, \"\", 9, fn_index=1)\n\n    def postprocess(self, output: str) -> str:\n        return [os.path.join(output, i) for i in os.listdir(output) if not i.endswith(\"json\")][0]\n\n    def _block_input(self, gr) -> \"gr.components.Component\":\n        return gr.Textbox()\n\n    def _block_output(self, gr) -> \"gr.components.Component\":\n        return gr.Image()\n```\n\nSome notes on this implementation:\n\n1. All instances of `GradioTool` have an attribute called `client` that is a pointed to the underlying [gradio client](https://github.com/gradio-app/gradio/tree/main/client/python#gradio_client-use-a-gradio-app-as-an-api----in-3-lines-of-python). That is what you should use\n   in the `create_job` method.\n2. `create_job` just passes the query string to the `submit` function of the client with some other parameters hardcoded, i.e. the negative prompt string and the guidance scale. We could modify our tool to also accept these values from the input string in a subsequent version.\n3. The `postprocess` method simply returns the first image from the gallery of images created by the stable diffusion space. We use the `os` module to get the full path of the image.\n\n## Conclusion\n\nYou now know how to extend the abilities of your LLM with the 1000s of gradio spaces running in the wild!\nAgain, we welcome any contributions to the [gradio_tools](https://github.com/freddyaboulton/gradio-tools) library.\nWe're excited to see the tools you all build!\n", "tags": [], "spaces": [], "url": "/guides/gradio-and-llm-agents/", "contributor": null}}