Integration & Unit Tests with TDD in Rails
Tags: ruby, rails, testing
Published: 2020-01-08 01:36:53

---
title: Integration & Unit Tests with TDD in Rails
published: true
description: Using Test Driven Development to build a new feature
tags: Ruby, Rails, Testing 
---

Recent Articles:
1. [How to fix "Posting failed. LinkedIn retry" error when posting on LinkedIn business page](https://socialscheduler.co/how-to-fix-post-failed-please-retry-linkedin-business-error/)
2. [Guide: How to schedule LinkedIn posts for free](https://socialscheduler.co/how-to-schedule-linkedin-posts/)

-

The advantage of using outside-in TDD is that it flows naturally from the business requirements, to the user actions, and then to the internals of the application - and we don't have to think too far ahead while implementing this. We just have to worry about the next thing in the test, and it will drive you to implement all the details, and then deliver the feature.

The challenge with TDD is that you have to be really familiar with Rails and testing in particular. Because right in the beginning you have to write the high-level integration tests which requires several technologies to work well together. And when the test fails, you have to know exactly what needs to be done to pass the test.

## Implementing a new feature via TDD

I have a Shopify app that is live on the app store, and for marketing purposes, I wanted to build a free feature for the app. This post looks into how I went about adding the new feature using Test Driven Development. 

This is the sample flow for the new feature:
1. A user visits the landing page. 
2. He then enters the main keyword and the Shopify product page URL on the form
3. When he submits the form, the details are saved in the database
4. A background-job is started in parallel (Lets call it ScrapePageWorker)
5. The user is taken to the results page

Unit tests are useful for testing business logic in models, and controller tests are useful for testing controllers in isolation - making sure they receive the right inputs and that they render the right response. The above 5 step flow seems ideal for an integration test as we are not dealing with just one controller. Instead we are going all over the application - starting with the landing page, talking to the database, starting a BG job and being redirected to another page.  

## Integration Tests

I am using [Rspec](https://github.com/rspec/rspec-rails) for testing, and I start by creating a new file `boostctr/spec/features/free_marketing_tool_spec.rb`

Lets start with the first step in sample work flow: _A user visits the landing page._

```ruby
require 'spec_helper'
  feature "free marketing tool" do
    scenario "first time" do 
      visit audits_path
      expect(page).to have_content("Shopify Product Page SEO Checker")
    end
  end
end
```


In the next two steps, he _enters the details into the form and submits it_. I am using [Capybara](https://github.com/teamcapybara/capybara) for simulation real-world interactions - to get the virtual user to fill in the details in the form. 

```diff
require 'spec_helper'
  feature "free marketing tool" do
    scenario "first time" do 
      visit audits_path
      expect(page).to have_content("Shopify Product Page SEO Checker")
+     fill_in "Enter focus keyword", with: "modular sofa"
+     fill_in "Enter page URL", with: "https://hauslondon.com/products/mags-modular-sofa-by-hay"
+     click_button "Check Product Page SEO"
    end
  end
end
```

Upto now, the user has visited the page, filled in the details and clicked the submit button. At this stage a few things should happen simultaneously - 1) The details are saved to the database, 2) the user is redirected to another page, and 3) a background job starts to scrape the product page to get all the required data. 

Let's start with the HTTP side by making sure that the _user is redirected to the show page_. 

```diff
require 'spec_helper'
  feature "free marketing tool" do
    scenario "first time" do 
      visit audits_path
      expect(page).to have_content("Shopify Product Page SEO Checker")
      fill_in "Enter focus keyword", with: "modular sofa"
      fill_in "Enter page URL", with: "https://hauslondon.com/products/mags-modular-sofa-by-hay"
      click_button "Check Product Page SEO"
+     expect(page).to have_content("Thank you! We are checking if the product page is SEO optimised.")
    end
  end
end
```

Next, we'll check the database to make sure that the _form details have been saved_ into a table called `Audits`, and that the _information stored is correct_. 

```diff
require 'spec_helper'
  feature "free marketing tool" do
    scenario "first time" do 
      visit audits_path
      expect(page).to have_content("Shopify Product Page SEO Checker")
      fill_in "Enter focus keyword", with: "modular sofa"
      fill_in "Enter page URL", with: "https://hauslondon.com/products/mags-modular-sofa-by-hay"
      click_button "Check Product Page SEO"
      expect(page).to have_content("Thank you! We are checking if the product page is SEO optimised.")
+     expect(Audit.first.focus_keyword).to eq('modular sofa')
+     expect(Audit.first.url).to eq('https://hauslondon.com/products/mags-modular-sofa-by-hay')
    end
  end
end
```

We also need to make sure that the _Sidekiq background job runs properly_. For this, we'll call the inline mode.

```diff
require 'spec_helper'
  feature "free marketing tool" do
    scenario "first time" do 
      visit audits_path
      expect(page).to have_content("Shopify Product Page SEO Checker")
      fill_in "Enter focus keyword", with: "modular sofa"
      fill_in "Enter page URL", with: "https://hauslondon.com/products/mags-modular-sofa-by-hay"
      click_button "Check Product Page SEO"
      expect(page).to have_content("Thank you! We are checking if the product page is SEO optimised.")
      expect(Audit.first.focus_keyword).to eq('modular sofa')
      expect(Audit.first.url).to eq('https://hauslondon.com/products/mags-modular-sofa-by-hay')
+     Sidekiq::Testing.inline! do
+       MarketingTool::ScrapePageWorker.perform_async(Audit.first.id, Audit.first.focus_keyword, Audit.first.url)
+     end
    end
  end
end
```


## Background Workers

Calling Sidekiq inline makes the job run immediately instead of enqueuing it. For this to work, we have to create a Sidekiq background worker, and I do this by creating a new file `boostctr/workers/marketing_tool/scrape_page_worker.rb`  

```ruby
class MarketingTool::ScrapePageWorker
  include Sidekiq::Worker
  sidekiq_options :queue => 'default', :retry => false

  def perform(audit_id, keyword, url)
    PageScraper.new(audit_id, keyword, url).start_scraping
  end
end
```


## Service Objects

`Scraping a page` does not really fit into a model or a controller. Also, I feel that I will be using this page scraping functionality in other parts of my app. Therefore, I can use Service Objects for this scenario. Service Objects are Plain Old Ruby Objects(PORO) that are used to execute one single thing, and I can create a new file for this on `boostctr/services/page_scraper.rb`

For actually scraping and reading the contents of the product page URL I am using  the [HTTP gem](https://github.com/httprb/http) and [Nokigiri](https://github.com/sparklemotion/nokogiri). 

```ruby
class PageScraper
  attr_reader :audit_id, :keyword, :url
  
  def initialize(audit_id, keyword, url)
    @audit_id = audit_id
    @keyword = keyword
    @url = url
  end
 
  def start_scraping
    begin
      response = HTTP.get(url)
    rescue => e
      raise e
    else
      if response != nil 
        code = response.code
        if code && code.to_s.start_with?('2')
          doc = Nokogiri::HTML(response.to_s)
          MarketingTool::CheckTitleTagWorker.perform_async(audit_id, doc.title, keyword)
          MarketingTool::CheckUrlWorker.perform_async(audit_id, get_handle_from_url(url), keyword, keyword)
          MarketingTool::CheckHeaderWorker.perform_async(audit_id, doc.search('h1').map(&:text), keyword)
          MarketingTool::CheckMetaDescriptionTagWorker.perform_async(audit_id, doc.at("meta[name='description']")['content'], keyword) if doc.at("meta[name='description']")
          MarketingTool::CheckKeywordsFirst100Worker.perform_async(audit_id, get_all_image_tags(doc), keyword)
          ...
        end
      end
    end
  end
  
  def get_handle_from_url(url)
    url.sub(/(\/)+$/,'').match(/\/([^\/]+)\/?$/)[1].gsub("-", " ")
  end
  
  def get_all_image_tags(doc)
    image_tags = []
    doc.css('img').each do |img_node|
      img_attributes = img_node.attributes.values # list of image attributes
      img_attributes.each do |attr|
        image_tags << [attr.value]
      end
    end
    image_tags
  end
end

```


The service object above scrapes the page, and gets all the important SEO details of the page. Each SEO test is sent to a different worker. For example, checking if the the page title has a number in it is done by the `CheckTitleHasNumberWorker`. By separating each task into a different worker, we are ensuring that each worker is small and does only one thing. 

```ruby
class MarketingTool::CheckTitleHasNumberWorker
  include Sidekiq::Worker
  sidekiq_options :queue => 'default'
  
  def perform(audit_id, title, keyword)
    audit = Audit.find(audit_id)
    title_tag_has_numbers = title.downcase =~ /\d/ ? true : false
    audit.update(title_tag_has_numbers: title_tag_has_numbers)
  end
end
```


## Unit Tests

This completes the whole 5 step user story we wrote in the beginning. But we also need to add Unit tests, to make the Audit model bulletproof - i.e. to make sure that all validations pass before it is saved to the database. I need to test a few validations:
- In a Shopify store, all product URLs have to be under the /products folder  
- The User has to enter both the URL and the focus keyword

For creating a unit test, I add a new file `boostctr/spec/models/audit_spec.rb`

```ruby
require 'spec_helper'

describe Audit do 
  it "URL must be a Shopofy product page" do 
    correct_audit = Audit.new(focus_keyword: 'sofas', url: 'exmaple.com/products/sofa')
    correct_audit.save
    expect(Audit.count).to eq(1)
    wrong_audit = Audit.new(focus_keyword: 'sofas', url: 'exmaple.com/sofa')
    wrong_audit.save
    expect(Audit.count).to eq(1)
  end
  it "saves itself" do 
    audit = Audit.new(focus_keyword: 'sofas', url: 'exmaple.com/products/sofa')
    audit.save
    expect(Audit.first).to eq(audit)
  end
  it "keyword must not be empty" do 
    audit = Audit.new(url: 'exmaple.com/products/sofa')
    audit.save
    expect(Audit.count).to eq(0)
  end
  it "URL must not be empty" do 
    audit = Audit.new(focus_keyword: 'sofas')
    audit.save
    expect(Audit.count).to eq(0)
  end
end
```

## Conclusion

This post looked at how we can do the following
* Using Test Driven Development to build a new feature
* Creating integration tests to test the flow across multiple controllers
* Adding unit tests to test business logic and validations
* Adding background workers to run two or more tasks simultaneously
* And adding a Service Object for logic that does not fit into a model or a controller.