Web Scraping for custom API
Tags: webscraping, httparty, beginners, nokogiri
Published: 2019-09-07 17:38:03

---
title: Web Scraping for custom API
published: true
description: My experience creating a Ruby on Rails Web Scraper
tags: #webscraping #httparty #beginner #nokogiri
cover_image: https://thepracticaldev.s3.amazonaws.com/i/z5mrqk7lkits6gy7flfx.png
---
### Intro
For my final project at the Flatiron School, I decided that I wanted to build a custom __Ski__ _resort_, _forecast_, and _snow report_ tracking app. I recently moved back to the Seattle area after spending the last four years in a remote Colorado ski town and decided to build something I would actually want to use.

The idea stemmed from my obsessive stalking of weather in different cities near Seattle as I try to plan for my weekend activities. Where am I going to find the warmest weather? Where will I have most sun? Most importantly, where can I avoid the rain? I keep track of this information within my weather app on my iPhone and decide what my plans are based on that incoming information. 

During the winter I like to do the exact same thing, but I'm browsing various websites checking snow reports, ski resort ticket prices, and weather forecasts. I decided to create an application that incorporated all of those information points that I value as a user where I could then "favorite" them and keep track of them on my own personal page. 

After planning out the details I went looking for online API's both free and paid ones. I quickly realized that no online resource was going to provide me with the exact data that I wanted, let alone get half of it and for a decent price so I decided to teach myself how to create a web scraper. 

### Scraping 
There is a great online resource I came across that guides the developer(you and I) through the process of creating a clean and efficient scraping tool by utilizing Ruby, Nokogiri, and HTTParty. The back-end of my project is coded in Ruby on Rails and I highly recommend watching this 30-minute video to create a basic, yet efficient scraper. 
![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/rfphixpyx5xqsxb7lm1b.png)
[Link to YouTube Video](https://www.youtube.com/watch?v=b3CLEUBdWwQ)
 
### Source of Data
I sourced my information from a public website, and since my projects are just for fun and for my portfolio I'm not going to run into any copyright issues. I decided to scrape 3 different pages off of _skiresort.info_ and limit my data to North American resorts. 

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/jrcmto3zaa955a39l4la.png)

I have linked my project at the bottom if anyone is interested in checking out my scraping file. It's located in _back_end_final/scraper.rb_ I believe I scraped almost 90 snow reports, 500ish forecast reports, and almost 1300 resorts for my project. By inspecting the website and targeting the specific id of the element I wanted to be scraped I was able to play around in the terminal until I had every bit of data, down the proper weather icon for the day in my database. 

### Diving Deep
If you look at the code, you'll notice that my final function __resort scraper__ is filled with ternary statements. 
![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/32ycsseakxp38ua78o5g.png)
One of the most difficult things that I came across in this process was dealing with incomplete data. _skiresort.info_ hosts so much data on their site that they are unable to keep the amounts available for each resort uniform. Some small ski resort in Canada simply cannot have the same amount of information about it as Whistler or Vail. I dealt with this problem by utilizing ternary statements and diving into the nth-child elements to target the exact data point I wanted for my application. 

It's pretty hacky looking and I should probably refactor it, but hey, it works!
Extracting my own data for a project was a lot of fun and definitely see myself using a Web Scraper for my upcoming projects. I realized that __Pow Tracker__ * only * scrapes data from the date that you run the scraping function. Since that makes for a pretty inefficient tracker of data, I would like to set a stretch goal for myself to automate the scraper so that I can have real-time data to work with. 
 
{% github https://github.com/sofiajonsson/back_end_final %}