Web Scraping Dynamic Websites With Scrapy Playwright
Tags: webscraping, scrapy, playwright, headlessbrowsers
Published: 2024-03-06 17:23:24

![Web Scraping Dynamic Web Pages With Scrapy Playwright](https://scrapfly.io/blog/content/images/web-scraping-dynamic-websites-with-scrapy-playwright_banner.svg)

Scrapy is a widely used web scraping library with convenient and comprehensive architecture support for the common web scraping processes. However, it lacks a major feature: JavaScript rendering.

In this tutorial, we'll explore Selenium Playwright. A Scrapy integration that allows scraping dynamic web pages with Scrapy. We'll explain web scraping with Scrapy Playwright through an example project and how to use it for common scraping use cases, such as clicking elements, scrolling and waiting for elements. Let's dive in!

## What is Scrapy Playwright?
<a href="https://github.com/scrapy-plugins/scrapy-playwright" target="_blank" rel="noopener noreferrer nofollow" class="shortcode-url">scrapy-playwright</a> is an integration between Scrapy and Playwright. It enables scraping dynamic web pages with Scrapy by __processing the web scraping requests using a Playwright instance__.

Scrapy Playwright allows for accessing the used Playwright pages, which enables most of the Playwright features such as:
- Simulating mouse and keyboard actions.
- Waiting for events, load states and HTML elements.
- Taking screenshots.
- Executing custom JavaScript code.

## How to Install Scrapy Playwright?
To web scrape with Scrapy Playwright, we'll have to install a few Python libraries:
- <a href="https://pypi.org/project/Scrapy/" target="_blank" rel="noopener noreferrer" class="shortcode-url">Scrapy</a>: For creating a Scrapy project and executing the scraping spiders.
- <a href="https://pypi.org/project/scrapy-playwright/" target="_blank" rel="noopener noreferrer" class="shortcode-url">scrapy-playwright</a>: A middleware for processing the requests using Playwright.
- <a href="https://pypi.org/project/playwright/" target="_blank" rel="noopener noreferrer" class="shortcode-url">Playwright</a>: The Playwright Python API for automating the headless browsers.

The above libraries can be installed using the `pip` command:
```shell
pip install scrapy scrapy-playwright playwright
```
After running the above command, install the Playwright headless browser binaries and dependencies:
```shell
playwright install chromium
playwright install-deps chromium
```
The above command will install the related __Chrome__ binaries. However, we can specify other browser engines: `firefox` or `webkit`.

> 🙋‍ Note that scrapy-playwright relies on the asyncio SelectorEventLoop. So, to use Scrapy Playwright on Windows, we have to use WSL. An interface for running Linux environments in Windows. For the installation instructions, refer to the official Microsoft guide.

## How to Scrape with Scrapy Playwright?
In this section, we'll go over a step-by-step tutorial on creating a Scrapy project, integrating it with Playwright and creating a scraping Spider to extract data using Playwright.

This Scrapy Playwright tutorial will briefly cover the basics of Scrapy. For further details, refer to our [dedicated guide on Scrapy](https://scrapfly.io/blog/web-scraping-with-scrapy/).

## Setting Up Scrapy Project
Let's start out with creating a new Scrapy project through the `Scrapy` commands:
```shell
$ scrapy startproject reviewgather reviewgather-scraper
#                     ^ name       ^ project directory
```

Executing the above command will create a Scrapy project in the `reviewgather-scraper` folder. Let's navigate to its directory and inspect the created project files:
```
$ cd reviewgather-scraper
$ tail
.
├── reviewgather
│   ├── __init__.py
│   ├── items.py
│   ├── middlewares.py
│   ├── pipelines.py
│   ├── settings.py 
│   └── spiders
│       ├── __init__.py 
└── scrapy.cfg
```

Now that the Scrapy project is ready. Let's power it with Playwright!

### Integrating Playwright With Scrapy
Setting Playwright with Scrapy is fairly straightforward. All we have to do is add these two lines to the `settings.py` file in the Scrapy project:
```python
DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}
```

Also, enable the `AsyncioSelectorReactor` by making sure that the following line exists in the same file and add it if not:
```python
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
```

Our Scrapy project can now use Playwright. Let's create the first Scrapy Playwright scraping spider to put it into evaluation!

## Creating Scraping Spider
In this Scrapy Playwright tutorial, we'll scrape review data from <a href="https://web-scraping.dev/testimonials" target="_blank" rel="noopener noreferrer nofollow" class="shortcode-url">web-scraping.dev</a>:
<figure class="kg-card kg-image-card kg-card-hascaption shortcode-img">
            <img class="kg-image medium-zoom-image" loading="lazy" src="https://scrapfly.io/blog/content/images/2024/02/web-scraping.dev-review-data-1.webp" alt="webpage with review data" title="" width="700" height="341" srcset="https://scrapfly.io/blog/content/images/size/w300/2024/02/web-scraping.dev-review-data-1.webp 300w,
https://scrapfly.io/blog/content/images/size/w600/2024/02/web-scraping.dev-review-data-1.webp 600w,
https://scrapfly.io/blog/content/images/size/w1000/2024/02/web-scraping.dev-review-data-1.webp 1000w,
https://scrapfly.io/blog/content/images/size/w1300/2024/02/web-scraping.dev-review-data-1.webp 1300w,
https://scrapfly.io/blog/content/images/size/w2000/2024/02/web-scraping.dev-review-data-1.webp 2000w,"><figcaption>Reviews on web-scraping.dev</figcaption>
            </figure>

To scrape the above review data, we have to create a Scrapy spider:
```shell
$ scrapy genspider reviews web-scraping.dev
#                  ^ name  ^ domain to scrape
```

The above Scrapy command will generate a spider named `reviews.py` with a boilerplate code:
```python
import scrapy


class ReviewsSpider(scrapy.Spider):
    name = "reviews"
    allowed_domains = ["web-scraping.dev"]
    start_urls = ["https://web-scraping.dev"]

    def parse(self, response):
        pass
```

The starting point of the above spider is `start_urls`, which is used for crawling purposes. Since our scraping target is only one page, we'll change it to a `start_requests` function and request the target web page with Playwright:
```python
import scrapy


class ReviewsSpider(scrapy.Spider):
    name = "reviews"
    allowed_domains = ["web-scraping.dev"]

    def start_requests(self):
        url = "https://web-scraping.dev/testimonials"
        yield scrapy.Request(
            url=url,
            meta={
                "playwright": True
            }
        )

    def parse(self, response):
        reviews = response.css("div.testimonial")
        for review in reviews:
            yield {
                "rate": len(review.css("span.rating > svg").getall()),
                "text": review.css("p.text::text").get()
            }
```

Let's go through the above spider changes:
- Add a `start_requests` function and along with the target page URL.
- Request the URL using the `scrapy.Request` method and add the `playwright` parameter to the request metadata to process it with Playwright.
- Update that `parse()` callback function to parse the review data on the page by iterating and extracting them using <a class="text-reference shortcode-tref" href="https://scrapfly.io/blog/parsing-html-with-css/">CSS selectors</a>.

The next step is executing the reviews spider and save the results:
```shell
scrapy crawl reviews --output reviews.json
```

The above command will create a `reviews.json` with the data extracted:
```json
[
    {"rate": 5, "text": "We've been using this utility for years - awesome service!"},
    {"rate": 5, "text": "This Python app simplified my workflow significantly. Highly recommended."},
    {"rate": 4, "text": "Had a few issues at first, but their support team is top-notch!"},
    {"rate": 5, "text": "A fantastic tool - it has everything you need and more."},
    {"rate": 5, "text": "The interface could be a little more user-friendly."},
    {"rate": 5, "text": "Been a fan of this app since day one. It just keeps getting better!"},
    {"rate": 4, "text": "The recent updates really improved the overall experience."},
    {"rate": 3, "text": "A decent web app. There's room for improvement though."},
    {"rate": 5, "text": "The app is reliable and efficient. I can't imagine my day without it now."},
    {"rate": 1, "text": "Encountered some bugs. Hope they fix it soon."}
]
```

Cool, our Scrapy Playwright scraping spider extracted the review data! However, it only contains the data from the first review page. To load and scrape more reviews, we have to scroll down the page. To do this, let's have a closer look at configuring Scrapy Playwright and automating the headless browser!

## Implement Common Scraping Cases With Scrapy Selenium
In the following sections, we'll explore configuring Playwright with the Scrapy setup and controlling the Plawright headless browser for common web scraping use cases.

The `scrapy-playwright` middleware supports most of the Playwright methods. This means that __we can apply the regular Playwright features in Scrapy__. For further details on these features, refer to our [dedicated guide on Playwright](https://scrapfly.io/blog/web-scraping-with-playwright-and-python/).

### Configuring Scrapy Playwright
Before we explore using Scrapy Playwright to execute different web scraping tasks, let's have a look at configuring the Playwright browser and its context first.

The scrapy-playwright middleware allows for defining global Playwright configuration through the `settings.py` file in the Scrapy project:
```python
#settings.py

PLAYWRIGHT_BROWSER_TYPE = "chromium"

PLAYWRIGHT_LAUNCH_OPTIONS = {
    "headless": False, # run in the headful mode
    "timeout": 60 * 1000,  # 60 seconds
}

PLAYWRIGHT_CONTEXTS = {
    "some_context_name": {
        "viewport": {"width": 1280, "height": 720},
        "locale": "fe-FR",
        "timezone_id": "Europe/Paris",
    }
}
```

In the above code, we define two different configurations:
- Launch options: A timeout for the browser instance and whether to run the browser in headless mode.
- Browser context: The Playwright browser <a href="https://playwright.dev/python/docs/emulation" target="_blank" rel="noopener noreferrer nofollow" class="shortcode-url">emulation</a> settings, such as viewport and <a class="text-reference shortcode-tref" href="https://scrapfly.io/blog/how-to-scrape-in-another-language-or-currency/">locality</a> configuration.

The Playwright context settings are global and can include several context profiles. They can be used by declaring the profile name in the request metadata:
```python
        yield scrapy.Request(
            url=url,
            meta={
                "playwright": True,
                "playwright_context": "some_context_name",
            }
        )
```

Scrapy Playwright also allows for defining custom Headers and Cookies that will be used across all the requests:
```Python
# settings.py

from playwright.async_api import Request
from scrapy.http.headers import Headers

def custom_headers(
    browser_type: str,
    playwright_request: Request,
    scrapy_headers: Headers,
) -> dict:
    return {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36 Edg/113.0.1774.35"}

PLAYWRIGHT_PROCESS_REQUEST_HEADERS = custom_headers
```

Here, we define a `custom_headers` function that returns specific headers values and pass it to `PLAYWRIGHT_PROCESS_REQUEST_HEADERS` to use it across all Playwright requests. __It will also override the default Scrapy headers and the headers passed to the Scrapy request__.

### Scrolling
Let's update our previous Scrapy Playwright scraping spider to scroll down and load more reviews. For this, we'll use the <a href="https://github.com/scrapy-plugins/scrapy-playwright#pagemethod-class" target="_blank" rel="noopener noreferrer" class="shortcode-url">scrapy-playwright PageMethod</a>, which supports most of the default <a href="https://playwright.dev/python/docs/api/class-page" target="_blank" rel="noopener noreferrer nofollow" class="shortcode-url">Playwright page methods</a>.

We'll execute a custom JavaScript code to simulate a scroll action to load all the review page data and then parse them:
```python
import scrapy
from scrapy_playwright.page import PageMethod


class ReviewsSpider(scrapy.Spider):
    name = "reviews"
    allowed_domains = ["web-scraping.dev"]

    def start_requests(self):
        url = "https://web-scraping.dev/testimonials"
        yield scrapy.Request(
            url=url,
            meta={
                "playwright": True,
                "playwright_page_methods": [
                    # execute the scroll script
                    PageMethod("evaluate", "for (let i = 0; i < 8; i++) setTimeout(() => window.scrollTo(0, document.body.scrollHeight), i * 2000);"),
                    # wait for 30 seconds
                    PageMethod("wait_for_timeout", 15000)
                ],
            }
        )

    async def parse(self, response):
        reviews = response.css("div.testimonial")
        for review in reviews:
            yield {
                "rate": len(review.css("span.rating > svg").getall()),
                "text": review.css("p.text::text").get()
            }
```

The above code is almost the same as the previous spider one. We only execute a JavaScript code for scrolling and wait for 15 seconds for the script to finish.

If we run the above spider and look at the result file, we'll find all the review data scraped:
```json
[
    ....
    {"rate": 5, "text": "I've tried many similar apps, but this one stands out with its exceptional performance and features."},
    {"rate": 2, "text": "The app's user interface is outdated and not intuitive. It needs a modern redesign."},
    {"rate": 5, "text": "I'm extremely satisfied with this app. It has exceeded my expectations in every way."},
    {"rate": 5, "text": "The app's documentation is comprehensive and easy to follow, making it easy to get started."},
    {"rate": 5, "text": "The app's performance has been flawless. I haven't experienced any issues or slowdowns."}    
]
```

We can successfully handle infinite scrolling with Scrapy Playwright. However, __the script waits for a fixed timeout, which isn't advised__. Let's wait for a specific element instead!

### Timeouts and Waiting For Elements
Playwright provides support for different waiting types:
- An event.
- A function to finish.
- A load state, either as `domcontentloaded` or `networkidle`.
- A URL, in case of navigation.
- A specific element to be present.
- Fixed timeouts.

__Relying on dynamic timeouts is more efficient in terms of performance__, as it reduces the unnecessary delays between the script actions.

The load state and fixed timeouts are usually used to wait for the natural page loading without explicit actions from the scraper side:
```python
    def start_requests(self):
        url = "https://web-scraping.dev/testimonials"
        yield scrapy.Request(
            url=url,
            meta={
                "playwright": True,
                "playwright_page_methods": [
                    # fixed timeout wait
                    PageMethod("wait_for_timeout", 5000),
                    # # wait for the document to load
                    PageMethod("wait_for_load_state", "domcontentloaded"),
                    # # wait for the network to be idle
                    PageMethod("wait_for_load_state", "networkidle"),
                ],
            }
        )
```

In the context of our reviews scraper, we'll wait for the latest review on the page to load after the scroll:
```python
class ReviewsSpider(scrapy.Spider):
    name = "reviews"
    allowed_domains = ["web-scraping.dev"]

    def start_requests(self):
        url = "https://web-scraping.dev/testimonials"
        yield scrapy.Request(
            url=url,
            meta={
                "playwright": True,
                "playwright_page_methods": [
                    # fixed timeout wait
                    PageMethod("evaluate", "for (let i = 0; i < 10; i++) setTimeout(() => window.scrollTo(0, document.body.scrollHeight), i * 2000);"),
                    # wait for latest element to load
                    PageMethod("wait_for_selector", "div.testimonial:nth-child(60)"),
                ],
            }
        )

    async def parse(self, response):
        reviews = response.css("div.testimonial")
        for review in reviews:
            yield {
                "rate": len(review.css("span.rating > svg").getall()),
                "text": review.css("p.text::text").get()
            }
```

Here, we use the same JavaScript code to scroll and add an additional `PageMethod` to wait for the latest review element to appear on the HTML.

### Taking Screenshots
To capture a screenshot with Scrapy Playwright, we can utilize the `screenshot` page method:
```python
class ReviewsSpider(scrapy.Spider):
    name = "reviews"
    allowed_domains = ["web-scraping.dev"]

    def start_requests(self):
        url = "https://web-scraping.dev/testimonials"
        yield scrapy.Request(
            url=url,
            meta={
                "playwright": True,
                "playwright_page_methods": [
                    PageMethod(
                        "screenshot",
                        path="screenshot.png",
                        full_page=True # whether to capture the whole page
                    ),
                ],
            }
        )
```

Here, we use the `screenshot` method in the `PageMethod` parameter to save it to the project directory. However, screenshots are usually captured after some browser actions. Luckily, __we can capture the screenshot from the callback function__ instead:
```python
class ReviewsSpider(scrapy.Spider):
    name = "reviews"
    allowed_domains = ["web-scraping.dev"]

    def start_requests(self):
        url = "https://web-scraping.dev/testimonials"
        yield scrapy.Request(
            url=url,
            meta={
                "playwright": True,
                "playwright_include_page": True
            }
        )

    async def parse(self, response):
        page = response.meta["playwright_page"]
        await page.screenshot(path="screenshot.png", full_page=True)
```

Here, we pass the Playwright `page` instance using the `playwright_include_page`. Then, we access it from the response metadata and use it to take a screenshot directly.

### Clicking Buttons And Filling Forms
Interacting the DOM elements on a page is commonly used while web scraping. In this Scrapy Playwright tutorial, we'll explain clicking buttons and filling forms by attempting to log in to the <a href="https://web-scraping.dev/login" target="_blank" rel="noopener noreferrer nofollow" class="shortcode-url">web-scraping.dev/login</a> example.

We'll create a Scrapy Playwright spider to __request the page URL, accept the cookies policy, fill in the login credentials, and then click the login button__:
```python
# spiders/login.py
# scrapy crawl login
import scrapy
from scrapy_playwright.page import PageMethod


class LoginSpider(scrapy.Spider):
    name = "login"
    allowed_domains = ["web-scraping.dev"]

    def start_requests(self):
        url = "https://web-scraping.dev/login?cookies="
        yield scrapy.Request(
            url=url,
            meta={
                "playwright": True,
                "playwright_page_methods": [
                    # wait the page to fully load
                    PageMethod("wait_for_load_state", "networkidle"),
                    # accept the cookie policy
                    PageMethod("click", "button#cookie-ok"),
                    # fill in the login creadentials
                    PageMethod("fill", "input[name='username']", "user123"),
                    PageMethod("fill", "input[name='password']", "password"),
                    # click submit button
                    PageMethod("click", "button[type='submit']"),
                    # wait for an element on the reidrect page
                    PageMethod("wait_for_selector", "div#secret-message"),
                ]
            }
        )

    def parse(self, response):
        print(f"The secret message is {response.css('div#secret-message::text').get()}")
        "The secret message is 🤫"
```  

Before you run the above spider, make sure to disable the default Scrapy headers by adding the following line to the settings.py file: `PLAYWRIGHT_PROCESS_REQUEST_HEADERS = None`

In the above Scrapy Playwright scraper, we use the `click` and `fill` to complete the login process while utilizing timeouts between the steps to ensure a successful execution.

## ScrapFly: Scrapy Playwright Alternative
ScrapFly is a web scraping API that supports scraping dynamic web pages using a <a href="https://scrapfly.io/docs/scrape-api/javascript-rendering" target="_blank" rel="noopener noreferrer" class="shortcode-url">JavaScript rendering</a> feature. It also provides built-in <a href="https://scrapfly.io/docs/scrape-api/javascript-scenario" target="_blank" rel="noopener noreferrer" class="shortcode-url">JavaScript scenarios</a> for controlling the headless browsers for common scraping use cases, such as waiting for elements, scrolling, filling and clicking elements.

Moreover, ScrapFly allows for scraping at scale by providing:
- <a href="https://scrapfly.io/docs/scrape-api/anti-scraping-protection" target="_blank" rel="noopener noreferrer" class="shortcode-url">Anti-scraping protection</a> bypass: For scraping any website without getting blocked.
- <a href="https://scrapfly.io/docs/scrape-api/proxy" target="_blank" rel="noopener noreferrer" class="shortcode-url">Residential proxies</a>s in over 50 countries: For avoiding <a class="text-reference shortcode-tref" href="https://scrapfly.io/blog/how-to-avoid-web-scraping-blocking-ip-addresses/">IP address blocking</a> and throttling while also allowing for scraping from almost any geographical location.
- <a href="https://scrapfly.io/docs/sdk/scrapy" target="_blank" rel="noopener noreferrer" class="shortcode-url">Scrapy Integration</a>, as well as <a href="https://scrapfly.io/docs/sdk/python" target="_blank" rel="noopener noreferrer" class="shortcode-url">Python</a> and <a href="https://scrapfly.io/docs/sdk/typescript" target="_blank" rel="noopener noreferrer" class="shortcode-url">Typescript</a> SDKs.
- <a href="https://scrapfly.io/docs/scrape-api/getting-started" target="_blank" rel="noopener noreferrer" class="shortcode-url">And much more!</a>

<figure class="kg-card kg-image-card kg-card-hascaption shortcode-img">
<img class="kg-image medium-zoom-image" loading="lazy" src="https://scrapfly.io/blog/content/images/common_scrapfly-api.svg" alt="scrapfly middleware" title="" width="600" height="241"><figcaption>ScrapFly service does the heavy lifting for you!</figcaption>
</figure>

ScrapFly is available as a Scrapy integration. Simply add the following lines to the `settings.py` file in the Scrapy project to authorize the API calls and set the concurrency limit:
```python
SCRAPFLY_API_KEY = "Your ScrapFly API key"
CONCURRENT_REQUESTS = 2  # Adjust according to your plan limit rate and your needs
```

Let's replicate the latest Scrapy spider with the ScrapFly API. All we have to do is enable the `asp` parameter to avoid scraping blocking and control the headless through the JavaScript scenarios.

ScrapFly X Scrapy:
```python
from scrapfly import ScrapeConfig
from scrapfly.scrapy import ScrapflyScrapyRequest, ScrapflySpider, ScrapflyScrapyResponse


class LoginSpider(ScrapflySpider):
    name = 'login'
    allowed_domains = ['web-scraping.dev']


    def start_requests(self):
        yield ScrapflyScrapyRequest(
            scrape_config=ScrapeConfig(
                # target website URL
                url="https://web-scraping.dev/login?cookies=",
                # bypass anti scraping protection
                asp=True,        
                # set the proxy location to a specific country
                country="US",
                # enable JavaScript rendering
                render_js=True,
                # scroll down the page automatically
                auto_scroll=True,
                # add JavaScript scenarios
                js_scenario=[
                    {"click": {"selector": "button#cookie-ok"}},
                    {"fill": {"selector": "input[name='username']","clear": True,"value": "user123"}},
                    {"fill": {"selector": "input[name='password']","clear": True,"value": "password"}},
                    {"click": {"selector": "form > button[type='submit']"}},
                    {"wait_for_navigation": {"timeout": 5000}}
                ],
                # take a screenshot
                screenshots={"logged_in_screen": "fullpage"}
            ),
            callback=self.parse
        )


    def parse(self, response: ScrapflyScrapyResponse):
        print(f"The secret message is {response.css('div#secret-message::text').get()}")
        "The secret message is 🤫"
```

ScrapFly SDK:
```python
from scrapfly import ScrapeConfig, ScrapflyClient, ScrapeApiResponse

scrapfly = ScrapflyClient(key="Your ScrapFly API key")

api_response: ScrapeApiResponse = scrapfly.scrape(
    ScrapeConfig(
        # target website URL
        url="https://web-scraping.dev/login?cookies=",
        # bypass anti scraping protection
        asp=True,        
        # set the proxy location to a specific country
        country="US",
        # # enable the cookies policy
        # headers={"cookie": "cookiesAccepted=true"},
        # enable JavaScript rendering
        render_js=True,
        # scroll down the page automatically
        auto_scroll=True,
        # add JavaScript scenarios
        js_scenario=[
            {"click": {"selector": "button#cookie-ok"}},
            {"fill": {"selector": "input[name='username']","clear": True,"value": "user123"}},
            {"fill": {"selector": "input[name='password']","clear": True,"value": "password"}},
            {"click": {"selector": "form > button[type='submit']"}},
            {"wait_for_navigation": {"timeout": 5000}}
        ],
        # take a screenshot
        screenshots={"logged_in_screen": "fullpage"},
        debug=True
    )
)

# get the HTML from the response
html = api_response.scrape_result['content']

# use the built-in Parsel selector
selector = api_response.selector
print(f"The secret message is {selector.css('div#secret-message::text').get()}")
"The secret message is 🤫"
```

<a href="https://scrapfly.io/register/" target="_blank" rel="noopener noreferrer" class="shortcode-url">Sign up</a> to get your API key!

## FAQ
To wrap up this guide on web scraping with Scrapy Playwright, let's have a look at some frequently asked questions.

#### How to solve the error "NotImplementedError in ('twisted.internet.asyncioreactor.AsyncioSelectorReactor')"?
This is a common error that occurs while running scrapy-playwright in Windows. It happens due to the lack of support for the `SelectorEventLoop` in Windows. The alternative for using Scrapy Playwright in Windows is running it on WSL. For further details, refer to the <a href="https://github.com/scrapy-plugins/scrapy-playwright#lack-of-native-support-for-windows" target="_blank" rel="noopener noreferrer" class="shortcode-url">official scrapy-playwright known issues</a>.

#### Can I scrape dynamic web pages with Scrapy?
Yes. Scrapy Playwright is a middleware integration that enables scraping dynamic pages with Scrapy by processing the requests using a Playwright instance.

#### Are there alternatives for Scrapy Playwright?
Yes, there are other integrations that allow Scrapy to scrape dynamic web pages, such as <a href="https://scrapfly.io/blog/web-scraping-dynamic-web-pages-with-scrapy-selenium/" target="_blank" rel="noopener noreferrer" class="shortcode-url">Scrapy Selenium</a> and <a href="https://scrapfly.io/blog/web-scraping-with-scrapy-splash/" target="_blank" rel="noopener noreferrer" class="shortcode-url">Scrapy Splash</a>.

## Summary

In this guide, we explored the scrapy-playwright integration, which allows scraping dynamic web pages with Scrapy using Playwright headless browsers.

We went through a step-by-step guide on installing Scrapy Playwright and using it through an example project. We have also explained implementing common web scraping with Scrapy Playwright use cases, such as:
- Handling infinite scrolling while scraping.
- Executing custom JavaScript code.
- Applying timeout waits.
- Taking screenshots.
- Clicking buttons and filling out forms.