Repository: langchain
Issue: Playwright Browser Freezing
Labels: Ɑ: agent,🤖:bug,stale

### System Info

## System Info

**LangChain Version:** 0.0.354  

**Platform:** MacOS Sonoma 14.2.1
**Python Version:** 3.11.6

### Who can help?

@hwchase17
@agola11

### Information

- [X] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [X] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

**Code sample that utilizes the Playwright Toolkit:** [https://python.langchain.com/docs/integrations/toolkits/playwright](https://python.langchain.com/docs/integrations/toolkits/playwright)

For instance, when you attempt to run:
```Python
await navigate_tool.arun(
    {"url": "https://web.archive.org/web/20230428131116/https://www.cnn.com/world"}
)
```
Nothing happens after 20 minutes.

The only way I have been able to get a response was by waiting a few seconds and then turning off my computer's wifi, which would return the expected:
```Bash
'Navigating to https://web.archive.org/web/20230428131116/https://www.cnn.com/world returned status code 200'
```

I am utilizing the browser in an agent, and here is the code:
```Python
import asyncio
from typing import Type, Optional

from langchain.agents import AgentExecutor
from langchain.schema import SystemMessage, HumanMessage, AIMessage
from long-chain.agents.format_scratchpad.openai_functions import format_to_openai_function_messages
from langchain.agents.output_parsers.openai_functions import OpenAIFunctionsAgentOutputParser
from langchain.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain.tools.convert_to_openai import format_tool_to_openai_function

from langchain_community.tools.playwright.utils import create_async_playwright_browser
from langchain.tools.tavily_search import TavilySearchResults
from langchain.utilities.tavily_search import TavilySearchAPIWrapper
from langchain_community.agent_toolkits.playwright import PlayWrightBrowserToolkit
from langchain.callbacks.manager import CallbackManagerForToolRun
from langchain.pydantic_v1 import BaseModel, Field
from langchain.tools import BaseTool

from auto_sentry.src.chat_models.enums.OpenAI import OpenAI as openai_enums
from auto_sentry.src.chat_models.Memory import Memory
from auto_sentry.src.chat_models.OpenAI import GPT
        
class AIQuestionInput(BaseModel):
    query: str = Field(description="should contain your question for the user including specific aspects or details from the user's input that require further clarification or elaboration")

class AIQuestionTool(BaseTool):
    name = "ai_question"
    description = "useful for when you need to clairify user requests or when you need to ask them a question"
    args_schema: Type[BaseModel] = AIQuestionInput

    def _run(
        self, query: str, run_manager: Optional[CallbackManagerForToolRun] = None
    ) -> str:
        """Use the tool."""
        return query


llm = GPT(model=openai_enums.GPT_4_1106_PREVIEW)
memory = Memory("Test_Conversation")
memory.clear()

search = [TavilySearchResults(api_wrapper=TavilySearchAPIWrapper(), handle_tool_error=True, verbose=True)]

playwright_async_browser = create_async_playwright_browser(headless=True)
toolkit = PlayWrightBrowserToolkit.from_browser(async_browser=playwright_async_browser)
playwright_tools = toolkit.get_tools()

tools:list = search + playwright_tools


MEMORY_KEY = "chat_history"
prompt = ChatPromptTemplate.from_messages(
    [
        SystemMessage(
            content="You are a helpful assistant."
        ),
        MessagesPlaceholder(variable_name=MEMORY_KEY),
        HumanMessage(content="{input}"),
        MessagesPlaceholder(variable_name="agent_scratchpad"),
    ]
)

memory.add(prompt.messages[0])

functions=[format_tool_to_openai_function(tool) for tool in tools]

#"""
llm_with_tools = llm.llm.bind(
    functions=functions
)

agent = (
    {
        "input": lambda x: x["input"],
        "agent_scratchpad": lambda x: format_to_openai_function_messages(
            x["intermediate_steps"]
        ),
        "chat_history": lambda x: x["chat_history"],
    }
    | prompt
    | llm_with_tools
    | OpenAIFunctionsAgentOutputParser()
)

agent_executer = AgentExecutor(
    agent=agent, tools=tools, verbose=True, return_intermediate_steps=True
)


query = HumanMessage(content=input("> "))
memory.add(query)

async def run_agent():
    print(memory.memory())
    response = await agent_executer.ainvoke(
        {"input": query, MEMORY_KEY: memory.memory()}
    )
    return response


response = asyncio.run(run_agent())

agent_messages = format_to_openai_function_messages(response["intermediate_steps"])
staged_messages = []
staged_messages.extend(agent_messages)
staged_messages.append(AIMessage(content=response["output"]))
memory.add(staged_messages)
print(memory.memory())
```

And when it executes a Playwright browser-related command, it just freezes and does nothing.

Here is an example that runs a Playwright browser-related command:
```Bash
> Summarize today's financial news from Google Finance.
[SystemMessage(content='You are a helpful assistant.'), HumanMessage(content="Summarize today's financial news from Google Finance.")]


> Entering new AgentExecutor chain...

Invoking: `navigate_browser` with `{'url': 'https://www.google.com/finance'}`
```

But when it utilizes any other tool, such as Tavily Search, it works successfully:
```Bash
> What is the weather supposed to be like in Miami tomorrow?            
[SystemMessage(content='You are a helpful assistant.'), HumanMessage(content='What is the weather supposed to be like in Miami tomorrow?')]


> Entering new AgentExecutor chain...

Invoking: `tavily_search_results_json` with `{'query': 'Miami weather forecast for tomorrow'}`


[{'url': 'https://www.weather.gov/mfl/\xa0', 'content': 'Read More > Privacy Policy Miami - South Florida Weather Forecast Office NWS Forecast Of
fice Miami - South Florida  Aviation Weather International Weather RADAR IMAGERY National Miami Radar Key West Radar Across Florida CLIMATE  Miami - South Florida11691 SW 17th StreetMiami, FL 33165305-229-4522Comments? Questions? Please Contact Us.  Last Map Update: Fri, Jan. 5, 2024 at 3:54:05 pm EST Text Product Selector: CURRENT HAZARDS OutlooksMiami - South Florida. Weather Forecast Office. NWS Forecast Office Miami - South Florida. Weather.gov > Miami ... Fri, Jan. 5, 2024 at 2:20:05 am EST. Watches, Warnings & Advisories. Zoom Out: Gale Warning: Small Craft Advisory: Rip Current Statement: ... National Weather Service Miami - South Florida 11691 SW 17th Street Miami, FL 33165 305 ...'}]The weather forecast for Miami tomorrow can be found on the National Weather Service's website for the Miami - South Florida region. You can visit [this link](https://www.weather.gov/mfl/) for the most up-to-date information on the weather forecast, including any watches, warnings, or advisories that may be in effect.                                                                                                                                            
> Finished chain.
[SystemMessage(content='You are a helpful assistant.'), HumanMessage(content='What is the weather supposed to be like in Miami tomorrow?'), AIMes
sage(content='', additional_kwargs={'function_call': {'arguments': '{"query":"Miami weather forecast for tomorrow"}', 'name': 'tavily_search_results_json'}}), FunctionMessage(content='[{"url": "https://www.weather.gov/mfl/\xa0", "content": "Read More > Privacy Policy Miami - South Florida Weather Forecast Office NWS Forecast Office Miami - South Florida  Aviation Weather International Weather RADAR IMAGERY National Miami Radar Key West Radar Across Florida CLIMATE  Miami - South Florida11691 SW 17th StreetMiami, FL 33165305-229-4522Comments? Questions? Please Contact Us.  Last Map Update: Fri, Jan. 5, 2024 at 3:54:05 pm EST Text Product Selector: CURRENT HAZARDS OutlooksMiami - South Florida. Weather Forecast Office. NWS Forecast Office Miami - South Florida. Weather.gov > Miami ... Fri, Jan. 5, 2024 at 2:20:05 am EST. Watches, Warnings & Advisories. Zoom Out: Gale Warning: Small Craft Advisory: Rip Current Statement: ... National Weather Service Miami - South Florida 11691 SW 17th Street Miami, FL 33165 305 ..."}]', name='tavily_search_results_json'), AIMessage(content="The weather forecast for Miami tomorrow can be found on the National Weather Service's website for the Miami - South Florida region. You can visit [this link](https://www.weather.gov/mfl/) for the most up-to-date information on the weather forecast, including any watches, warnings, or advisories that may be in effect.")]                                        
```

### Expected behavior

The behavior of the Playwright Browser should return results to an input in a relatively prompt manner, but it currently freezes and returns.