docs/md_v2/advanced/magic-mode.md

Magic Mode & Anti-Bot Protection

Crawl4AI provides powerful anti-detection capabilities, with Magic Mode being the simplest and most comprehensive solution.

Magic Mode

The easiest way to bypass anti-bot protections:

async with AsyncWebCrawler() as crawler:
    result = await crawler.arun(
        url="https://example.com",
        magic=True  # Enables all anti-detection features
    )

Magic Mode automatically:

• Masks browser automation signals
• Simulates human-like behavior
• Overrides navigator properties
• Handles cookie consent popups
• Manages browser fingerprinting
• Randomizes timing patterns

Manual Anti-Bot Options

While Magic Mode is recommended, you can also configure individual anti-detection features:

result = await crawler.arun(
    url="https://example.com",
    simulate_user=True,        # Simulate human behavior
    override_navigator=True    # Mask automation signals
)

Note: When magic=True is used, you don't need to set these individual options.

Example: Handling Protected Sites

async def crawl_protected_site(url: str):
    async with AsyncWebCrawler(headless=True) as crawler:
        result = await crawler.arun(
            url=url,
            magic=True,
            remove_overlay_elements=True,  # Remove popups/modals
            page_timeout=60000            # Increased timeout for protection checks
        )
        
        return result.markdown if result.success else None