Movie Crawler: Scraping 100,000+ Movie Information
Published: 2020-07-20 06:50:12

The data of movies record audiences' preferences and their attitude towards certain things. Gathering the movie info from relative websites, like IMDb and Rotten Tomatoes, will contribute to data analysis and data mining in the film industry. Generally speaking, the scraped data can be employed in some scenario:

· Analyzing the features of the target audience
· Obtaining public opinions to predict the coming trends
· Helping the Advertising Push
 
There are still more things that we can do with the movie data according to the needs. To help you fulfill data gathering, this article will introduce how to scrape the information from the IMDb Horror movie list, including director information, the cast of actors, and some other important information.

In this case, I’ll show you how to scrape the **134,555 Horror movie information** from IMDb, using the link:

[https://www.imdb.com/search/title/?genres=horror&start=51&explore=title_type,genres&ref_=adv_nxt](https://www.imdb.com/search/title/?genres=horror&start=51&explore=title_type,genres&ref_=adv_nxt)

The goal of this web scraper is to find films that are listed on the Horror movie list, obtaining director information, the cast of actors, and some other important information.

Before getting started, please [download Octoparse V7](https://www.octoparse.com/download) on your computer to follow up. Besides, it’s highly recommended to learn the [basic logic of using Octoparse](https://helpcenter.octoparse.com/hc/en-us/articles/900000659063-Lesson-0-Octoparse-Basics).

Let’s get started

**Step 1: Open the target website in the Octoparse built-in browser.**

Simply click “+task” under the Advanced Mode.
![Advanced Mode](https://www.octoparse.com/media/7691/image7.png)

Then, paste the URL to the box and click the “Save URL” button.
![Save URL](https://www.octoparse.com/media/7695/image11.png)

**Step 2: Click to build a task to scrape the movie information.**
After having the RUL opened in the Octoparse built-in browser, we can continue to build a pagination and a loop item to get the data.

Simply click the “next>>” element in the built-in browser and then click “Loop click selected element” on the Action Tips.

![Action Tips Penal](https://www.octoparse.com/media/7697/image2.png)

We can see the pagination has been built in the workflow.
![Pagination](https://www.octoparse.com/media/7699/image4.png)


If you want to make the Octoparse recognize the element you selected more precisely, you could simply [revise the XPath](https://helpcenter.octoparse.com/hc/en-us/articles/360041118892-What-is-XPath-and-how-to-use-it-in-Octoparse). As we can see in the below picture, the XPath that Octoparse generated is //DIV[@class='nav']/DIV[2]/A[2]. We’d better change it to //a[contains(text(), "Next »")]
![XPath](https://www.octoparse.com/media/7700/image5.png)

In this case, we need to scrape the data from the movie list, which says, we can directly create a loop item to extract the data.

Select one of the “blocks” on the browser, Octoparse can detect all the data fields in the blog you selected.

![Click to select](https://www.octoparse.com/media/7692/image8.png)

Then, select “Select all sub-elements”.

All the needed data are being selected by Octoparse and highlighted in red. Select “Select All” to continue.

![Click to select Info section](https://www.octoparse.com/media/7696/image1.png)

Finally, we select “Extract data in the loop”.

![Select the matching action](https://www.octoparse.com/media/7698/image3.png)

Now, we have both the pagination and the loop item done in Octoparse. We can see the workflow of the task on the left side and the data that are displayed on the right side.
![Data preview](https://www.octoparse.com/media/7694/image10.png)

**Step 3: Clean the data in Octoparse.**

Before extracting data, we’d better clean the data to make our final result better. Simple need to click to delete the unwanted field and rename the description you need.


**Step 4: Extract data**
Simply click “Extract data” to get the data locally.

![Extract data](https://www.octoparse.com/media/7693/image9.png)

As local extraction utilizes your own computer resources, such as the CPU, internet speed, it works slower than using Octoparse cloud extraction.

Anyway, after creating the scraper, what you need to do is wait and get the data, more than 100,000 lines of movie data in about 2 hours.

![final result](https://www.octoparse.com/media/7690/image6.png)

With the above steps, I suppose, everyone, including those who have no programming background can easily build a movie crawler with Octoparse V7 and get more than 100,000 lines of the movie information. However, that's not the easiest way. Using Octoparse V8 could be much easier:

![Octoparse 8: Auto-detection](https://www.octoparse.com/media/7701/op8-auto-detection.gif)

All in all, with data scraping, we can obtain the movie data online with any legal issue.

Apart from the data, the more important is about the skill you learned, which is extremely useful for doing the market research, keeping yourself updating, and many other things.