Repository: phidatahq/phidata
File: libs/agno/agno/knowledge/reader/website_reader.py (Python)

Code before:

        Parameters:
        - url (str): The starting URL to begin the crawl.
        - starting_depth (int, optional): The starting depth level for the crawl. Defaults to 1.

        Returns:
        - Dict[str, str]: A dictionary where each key is a URL and the corresponding value is the main
                          content extracted from that URL.

        Raises:
        - httpx.HTTPStatusError: If there's an HTTP status error.
        - httpx.RequestError: If there's a request-related error (connection, timeout, etc).

        Note:
        The function focuses on extracting the main content by prioritizing content inside common HTML tags
        like `<article>`, `<main>`, and `<div>` with class names such as "content", "main-content", etc.
        The crawler will also respect the `max_depth` attribute of the WebCrawler class, ensuring it does not
        crawl deeper than the specified depth.
        """
        num_links = 0
        crawler_result: Dict[str, str] = {}
        primary_domain = self._get_primary_domain(url)
        # Add starting URL with its depth to the global list
        self._urls_to_crawl.append((url, starting_depth))
        while self._urls_to_crawl:
            # Unpack URL and depth from the global list
            current_url, current_depth = self._urls_to_crawl.pop(0)

            # Skip if
            # - URL is already visited
            # - does not end with the primary domain,
            # - exceeds max depth
            # - exceeds max links
            if (
                current_url in self._visited
                or not urlparse(current_url).netloc.endswith(primary_domain)
                or (current_depth > self.max_depth and current_url != url)
                or num_links >= self.max_links
            ):
                continue

            self._visited.add(current_url)
            self.delay()

            try:
                log_debug(f"Crawling: {current_url}")

                response = (
                    httpx.get(current_url, timeout=self.timeout, proxy=self.proxy, follow_redirects=True)
                    if self.proxy
                    else httpx.get(current_url, timeout=self.timeout, follow_redirects=True)

Review comment (bug):
## Bug 3: WebsiteReader — sync crawl leaks state, async doesn't

Both `crawl()` and `async_crawl()` were added in the same commit (1f595c912). From day 1, `async_crawl()` (line 280) properly reset state:
```python
self._visited = set()
self._urls_to_crawl = [(url, starting_depth)]
```

But sync `crawl()` only did `self._urls_to_crawl.append(...)` — it appended to whatever was left over from the previous crawl.

**Before:**
```
reader.crawl("https://site-a.com")
  → _visited = {site-a.com, site-a.com/page1}
  → works fine on first call

reader.crawl("https://site-b.com")
  → _visited still contains site-a URLs
  → _urls_to_crawl.append(site-b.com) onto whatever was left
  → if site-b links to site-a.com, those links are skipped
```

**After:** Reset both `_visited` and `_urls_to_crawl` at the start, matching `async_crawl()` behavior exactly.

Code after:

        Parameters:
        - url (str): The starting URL to begin the crawl.
        - starting_depth (int, optional): The starting depth level for the crawl. Defaults to 1.

        Returns:
        - Dict[str, str]: A dictionary where each key is a URL and the corresponding value is the main
                          content extracted from that URL.

        Raises:
        - httpx.HTTPStatusError: If there's an HTTP status error.
        - httpx.RequestError: If there's a request-related error (connection, timeout, etc).

        Note:
        The function focuses on extracting the main content by prioritizing content inside common HTML tags
        like `<article>`, `<main>`, and `<div>` with class names such as "content", "main-content", etc.
        The crawler will also respect the `max_depth` attribute of the WebCrawler class, ensuring it does not
        crawl deeper than the specified depth.
        """
        num_links = 0
        crawler_result: Dict[str, str] = {}
        primary_domain = self._get_primary_domain(url)

        # Clear state so URLs from previous crawls aren't incorrectly skipped (matches async_crawl)
        self._visited = set()
        self._urls_to_crawl = [(url, starting_depth)]
        while self._urls_to_crawl:
            # Unpack URL and depth from the global list
            current_url, current_depth = self._urls_to_crawl.pop(0)

            # Skip if
            # - URL is already visited
            # - does not end with the primary domain,
            # - exceeds max depth
            # - exceeds max links
            if (
                current_url in self._visited
                or not urlparse(current_url).netloc.endswith(primary_domain)
                or (current_depth > self.max_depth and current_url != url)
                or num_links >= self.max_links
            ):
                continue

            self._visited.add(current_url)
            self.delay()

            try:
                log_debug(f"Crawling: {current_url}")

                response = (
                    httpx.get(current_url, timeout=self.timeout, proxy=self.proxy, follow_redirects=True)