Jordan Scrapes Redfin
Tags: webscraping, axios, cheerio, javascript
Published: 2020-10-22 21:37:27

---
title: Jordan Scrapes Redfin
published: true
date: 2020-10-19 14:22:00 UTC
tags: WebScraping,Axios,cheerio,Javascript
canonical_url: https://javascriptwebscrapingguy.com/jordan-scrapes-redfin/?utm_source=rss&utm_medium=rss&utm_campaign=jordan-scrapes-redfin
---

[Demo code here](https://github.com/aarmora/jordan-scrapes-redfin)

<iframe title="Scraping Redfin.com for property price" width="640" height="360" src="https://www.youtube.com/embed/ptU_BLX67js?feature=oembed" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<figcaption>Video walk through</figcaption>

Today we go to [Redfin](https://www.redfin.com/)! This is in the real estate data arena. It is the kind of thing that goes in pair with the post I wrote about scraping [real estate auctions](https://javascriptwebscrapingguy.com/jordan-scrapes-real-foreclose/). You would get the auction you are looking for and then go to Redfin.com to get some estimated pricing and other data.

## Investigation

![fun real estate gif](https://media.giphy.com/media/1g0JrJe5a6XsL0kMUk/giphy.gif)

When scraping a real estate site like this there are really two steps. The first is to be able to leverage an address to find the details page on the site. The second part is more obvious, just scraping the site for the desired data.

Redfin is a modern site and they do live returns of property information _as you type_. When they do this, it returns something that allows the user to go directly to the details page of this address. This almost for sure means that we can use it to find a way to the details page.

Check it.

[![web scraping redfin's search](https://javascriptwebscrapingguy.com/wp-content/uploads/2020/10/image-9-1024x215.png)](https://javascriptwebscrapingguy.com/wp-content/uploads/2020/10/image-9.png)

On the left you can see the searched data and the exact property discovered. On the right you can see the XHR requests that return the following data:

```
{}&&{"version":348,"errorMessage":"Success","resultCode":0,"payload":{"sections":[{"rows":[{"id":"1_60647192","type":"1","name":"3950 Callahan Dr","subName":"Memphis, TN, USA","url":"/TN/Memphis/3950-Callahan-Dr-38127/home/60647192","active":true,"claimedHome":false,"invalidMRS":false,"businessMarketIds":[58],"countryCode":"US"}],"name":"Addresses"}],"exactMatch":{"id":"1_60647192","type":"1","name":"3950 Callahan Dr","subName":"Memphis, TN, USA","url":"/TN/Memphis/3950-Callahan-Dr-38127/home/60647192","active":true,"claimedHome":false,"invalidMRS":false,"businessMarketIds":[58],"countryCode":"US"},"extraResults":{},"responseTime":0,"hasFakeResults":false,"isGeocoded":false,"isRedfinServiced":false}}
```

This data is kind of funny because it’s not quite JSON. Remove that first `{}&&` and the rest is valid JSON. And inside…we see a url! Bingo. We’re in business.

With this url, we can go directly to the webpage we are looking for. At the top, what do we find? The property value that we were looking for!

[![Redfin estimate.](https://javascriptwebscrapingguy.com/wp-content/uploads/2020/10/image-10.png)](https://javascriptwebscrapingguy.com/wp-content/uploads/2020/10/image-10.png)

Unfortunately, the details page doesn’t have any XHR requests with property data. The easiest way to confirm this is by looking at the network tab in developer tools and checking the “Doc” tab. If you see the page requested fully rendered then that means it is returning from the server fully fleshed out already.

![scraping Redfin Doc page](https://javascriptwebscrapingguy.com/wp-content/uploads/2020/10/image-11.png)

I’ll just use [cheerio](https://cheerio.js.org/) for this part and parse the HTML to get the price I’m looking for.

## The Code

![fun real estate gif](https://media.giphy.com/media/26tjZXz2YcvvjMVY4/giphy.gif)

Pretty simple code execution here. The async block that will handle it all will look like this:

```
const exampleAddresses = [
    '3950 CALLAHAN DR, Memphis, TN 38127',
    '17421 Deforest Ave, Cleveland, OH 44128',
    '1226 DIVISION AVENUE, San Antonio, TX 78225'
];

(async () => {

    for (let i = 0; i < exampleAddresses.length; i++) {
        const path = await getUrl(exampleAddresses[i]);

        console.log('path', path);

        const price = await getPrice(path);

        console.log('price', price);

        await timeout(2000);
    }
})();
```

You’d loop through your target addresses, get the url (really the path), and use that when you get the price.

```
async function getUrl(address: string) {
    // Location and v are required query parameters
    const url = `https://www.redfin.com/stingray/do/location-autocomplete?location=${address}&v=2`;

    const axiosResponse = await axios.get(url);

    const parsedData = JSON.parse(axiosResponse.data?.replace('{}&&', ''));

    return parsedData.payload.exactMatch.url;
}
```

The above function will get the path from that weird _almost_ JSON. We just get the data and then remove the `{}&&` with a replace function.

The `getPrice` function is a simple call with [axios](https://github.com/axios/axios) and parse with [cheerio](https://cheerio.js.org/).

```
async function getPrice(path: string) {
    const url = `https://redfin.com${path}`;

    const axiosResponse = await axios.get(url);

    const $ = cheerio.load(axiosResponse.data);

    let price = $('[data-rf-test-id="avm-price"] .statsValue').text();

    if (!price) {
        price = $('[data-rf-test-id="avmLdpPrice"] .value').text();
    }

    return price;
}
```

Bam. And that’s the end. We got ourselves some property prices from Redfin.

[Demo code here](https://github.com/aarmora/jordan-scrapes-redfin)

## Looking for business leads?

Using the techniques talked about here at [javascriptwebscrapingguy.com](https://javascriptwebscrapingguy.com), we’ve been able to launch a way to access awesome web data. Learn more at [Cobalt Intelligence](https://cobaltintelligence.com)!

The post [Jordan Scrapes Redfin](https://javascriptwebscrapingguy.com/jordan-scrapes-redfin/) appeared first on [Javascript Web Scraping Guy](https://javascriptwebscrapingguy.com).