Repository: bbc/wraith

Issue #401: `spider_skips` property broken
Created: 2016-03-14T02:11:11Z
I'm using the default spider.yaml from https://raw.githubusercontent.com/BBC-News/wraith/master/templates/configs/spider.yaml
I've tried running it with wraith installed locally on my mac, and also via the wraith docker image. Both fail, with different error messages.
On my mac locally:
$ wraith capture spider.yaml
Config validated. No serious issues found.
no paths defined in config, crawling from site root
creating new spider file
/Users/paullew/.rvm/gems/ruby-2.2.1/gems/anemone-0.7.2/lib/anemone/core.rb:298:in `=~': type mismatch: String given (TypeError)
    from /Users/paullew/.rvm/gems/ruby-2.2.1/gems/anemone-0.7.2/lib/anemone/core.rb:298:in `block in skip_link?'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/gems/anemone-0.7.2/lib/anemone/core.rb:298:in `any?'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/gems/anemone-0.7.2/lib/anemone/core.rb:298:in `skip_link?'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/gems/anemone-0.7.2/lib/anemone/core.rb:256:in `visit_link?'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/gems/anemone-0.7.2/lib/anemone/core.rb:151:in `block in run'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/gems/anemone-0.7.2/lib/anemone/core.rb:151:in `delete_if'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/gems/anemone-0.7.2/lib/anemone/core.rb:151:in `run'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/gems/anemone-0.7.2/lib/anemone/core.rb:92:in `block in crawl'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/gems/anemone-0.7.2/lib/anemone/core.rb:83:in `initialize'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/gems/anemone-0.7.2/lib/anemone/core.rb:90:in `new'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/gems/anemone-0.7.2/lib/anemone/core.rb:90:in `crawl'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/gems/anemone-0.7.2/lib/anemone/core.rb:18:in `crawl'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/gems/wraith-3.1.0/lib/wraith/spider.rb:69:in `spider'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/gems/wraith-3.1.0/lib/wraith/spider.rb:35:in `determine_paths'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/gems/wraith-3.1.0/lib/wraith/spider.rb:23:in `check_for_paths'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/gems/wraith-3.1.0/lib/wraith/cli.rb:36:in `check_for_paths'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/gems/wraith-3.1.0/lib/wraith/cli.rb:133:in `block in capture'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/gems/wraith-3.1.0/lib/wraith/cli.rb:28:in `within_acceptable_limits'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/gems/wraith-3.1.0/lib/wraith/cli.rb:130:in `capture'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/gems/thor-0.19.1/lib/thor/command.rb:27:in `run'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/gems/thor-0.19.1/lib/thor/invocation.rb:126:in `invoke_command'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/gems/thor-0.19.1/lib/thor.rb:359:in `dispatch'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/gems/thor-0.19.1/lib/thor/base.rb:440:in `start'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/gems/wraith-3.1.0/bin/wraith:5:in `<top (required)>'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/bin/wraith:23:in `load'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/bin/wraith:23:in `<main>'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/bin/ruby_executable_hooks:15:in `eval'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/bin/ruby_executable_hooks:15:in `<main>'

Running it via the wraith docker image:
$ docker run --rm -P -v ~/devel/resources/testing/wraith:/wraithy -w='/wraithy' bbcnews/wraith capture spider.yaml
/usr/local/lib/ruby/gems/2.1.0/gems/wraith-3.1.2/lib/wraith/spider.rb:64:in `spider': undefined local variable or method `wraith' for #<Wraith::Crawler:0x005648d0ca4be8> (NameError)
    from /usr/local/lib/ruby/gems/2.1.0/gems/wraith-3.1.2/lib/wraith/spider.rb:36:in `determine_paths'
    from /usr/local/lib/ruby/gems/2.1.0/gems/wraith-3.1.2/lib/wraith/spider.rb:24:in `check_for_paths'
    from /usr/local/lib/ruby/gems/2.1.0/gems/wraith-3.1.2/lib/wraith/cli.rb:36:in `check_for_paths'
    from /usr/local/lib/ruby/gems/2.1.0/gems/wraith-3.1.2/lib/wraith/cli.rb:134:in `block in capture'
    from /usr/local/lib/ruby/gems/2.1.0/gems/wraith-3.1.2/lib/wraith/cli.rb:28:in `within_acceptable_limits'
    from /usr/local/lib/ruby/gems/2.1.0/gems/wraith-3.1.2/lib/wraith/cli.rb:131:in `capture'
    from /usr/local/lib/ruby/gems/2.1.0/gems/thor-0.19.1/lib/thor/command.rb:27:in `run'
    from /usr/local/lib/ruby/gems/2.1.0/gems/thor-0.19.1/lib/thor/invocation.rb:126:in `invoke_command'
    from /usr/local/lib/ruby/gems/2.1.0/gems/thor-0.19.1/lib/thor.rb:359:in `dispatch'
    from /usr/local/lib/ruby/gems/2.1.0/gems/thor-0.19.1/lib/thor/base.rb:440:in `start'
    from /usr/local/lib/ruby/gems/2.1.0/gems/wraith-3.1.2/bin/wraith:5:in `<top (required)>'
    from /usr/local/bin/wraith:23:in `load'
    from /usr/local/bin/wraith:23:in `<main>'
Config validated. No serious issues found.
no paths defined in config, crawling from site root

Duplicate issue #422: Spider mode not working with wraith
Created: 2016-04-14T06:04:39Z
Hi all,
I could not get wraith working under spider mode.
Standard capture mode works fine.
ubuntu@ip-10-224-91-168:~/wraith$ wraith capture configs/carsalesspider.yaml
DEBUG: #################################################
DEBUG:   Wraith version:     3.1.4
DEBUG:   Ruby version:       ruby 1.9.3p484 (2013-11-22 revision 43786) [x86_64-linux]

DEBUG:   ImageMagick:        Version: ImageMagick 6.7.7-10 2014-03-06 Q16 http://www.imagemagick.org

DEBUG:   PhantomJS version:  2.1.1

DEBUG:   CasperJS version:   Couldn't find nor compute phantom.casperPath, exiting.

DEBUG: #################################################
DEBUG:
Config validated. No serious issues found.
no paths defined in config, crawling from site root
/var/lib/gems/1.9.1/gems/wraith-3.1.4/lib/wraith/spider.rb:64:in `spider': undefined local variable or method `wraith' for #<Wraith::Crawler:0x00000000cc9ef8> (NameError)
        from /var/lib/gems/1.9.1/gems/wraith-3.1.4/lib/wraith/spider.rb:36:in `determine_paths'
        from /var/lib/gems/1.9.1/gems/wraith-3.1.4/lib/wraith/spider.rb:24:in `check_for_paths'
        from /var/lib/gems/1.9.1/gems/wraith-3.1.4/lib/wraith/cli.rb:36:in `check_for_paths'
        from /var/lib/gems/1.9.1/gems/wraith-3.1.4/lib/wraith/cli.rb:134:in `block in capture'
        from /var/lib/gems/1.9.1/gems/wraith-3.1.4/lib/wraith/cli.rb:28:in `within_acceptable_limits'
        from /var/lib/gems/1.9.1/gems/wraith-3.1.4/lib/wraith/cli.rb:131:in `capture'
        from /var/lib/gems/1.9.1/gems/thor-0.19.1/lib/thor/command.rb:27:in `run'
        from /var/lib/gems/1.9.1/gems/thor-0.19.1/lib/thor/invocation.rb:126:in `invoke_command'
        from /var/lib/gems/1.9.1/gems/thor-0.19.1/lib/thor.rb:359:in `dispatch'
        from /var/lib/gems/1.9.1/gems/thor-0.19.1/lib/thor/base.rb:440:in `start'
        from /var/lib/gems/1.9.1/gems/wraith-3.1.4/bin/wraith:5:in `<top (required)>'
        from /usr/local/bin/wraith:23:in `load'
        from /usr/local/bin/wraith:23:in `<main>'

Config file info:
##############################################################
##############################################################
# This is an example configuration provided by Wraith.
# Feel free to amend for your own requirements.
# ---
# This particular config is intended to demonstrate how
# to use Wraith in 'spider' mode.
##############################################################
##############################################################


# Add as many domains as necessary. Key will act as a label
domains:
  my_site:       "http://www.carsales.com"
  my_other_site: "http://www.carsales.com"

# Notice the absence of a `paths` property. When no paths are provided, Wraith defaults to
# spidering mode to check your entire website.

# A list of URLs to skip when spidering.
# Ruby regular expressions can be used, if prefixed with `!ruby/regexp` as defined in the YAML Cookbook<200b>.
# See http://www.yaml.org/YAML_for_ruby.html#regexps
spider_skips:
#  - /foo/bar.html           # Matches /foo/bar.html explicitly
#  - !ruby/regexp /^\/baz\// # Matches any URLs that start with /baz

# the filename of the spider file to use. Default: spider.txt
spider_file: example_com_spider.txt

# the number of days to keep the site spider file
spider_days: 10

# amount of fuzz ImageMagick will use when comparing images. A higher fuzz makes the comparison less strict.
fuzz: '20%'

# the maximum acceptable level of difference (in %) between two images.
# Wraith considers it a failure if an image diff goes above this threshold.
threshold: 5

# screen widths (and optional height) to resize the browser to before taking the screenshot
screen_widths:
  - 1280

# the engine to run Wraith with.
browser: "phantomjs"

# the directory that your latest screenshots will be stored in
directory: 'www/csspider'

# choose how results are displayed in the gallery (default is `alphanumeric` if omitted)
# Different screen widths are always grouped together.
# Options:
#   alphanumeric - all paths (with or without a difference) are shown, sorted by path
#   diffs_first - all paths (with or without a difference) are shown, sorted by difference size (largest first)
#   diffs_only - only paths with a difference are shown, sorted by difference size (largest first)
mode: diffs_first

verbose: true
Cheers,
Ming Zhu

Closing comment (2016-04-15T15:41:07Z): Duplicate of #401. Thanks for the detailed config and output information though.