How SerpApi sped up data extraction from HTML from 3s to 800ms (or How to profile and optimize Ruby code and C extension)
Tags: ruby, performance, profiling, scraping
Published: 2021-02-02 12:21:13

---
canonical_url: https://serpapi.com/blog/data-extraction-speed-up-3s-to-800ms/
description: "Optimization of Ruby C extension with help of Linux perf, gdb, pprof, rbspy, and kind people."
---

In October 2020, our customers [reported response time inflation][4] for our [API to scrape Google Shopping Results][5]. We looked into it and sped up our parsers three times by shipping two pull requests to Nokogiri, trying to patch `libxml2`, and improving algorithms in our code. Below I describe our way from problem to solutions, and how to improve it further.

**TL;DR.** Use flame graphs to detect performance bottlenecks and verify improvements.

## HTML parsing speedup

Initially, we aggregated Logflare logs and looked into individual responses. Then we profiled our code and benchmarked assumptions.

### Logs

We reported `2.23s` as `total_time_taken` but customer reports `5.3577799797058105s`, Rails reports `5.129043s` (`x_runtime`), and Cloudflare reports `5.194s` (`origin_time`). Three seconds difference is huge. Probably it was coming from HTML to JSON parsing as it's the main thing not included in our JSON `total_time_taken`.

![50K responses per day took more than 10 seconds according to Logflare
](https://dev-to-uploads.s3.amazonaws.com/i/4qj1r7yhcitzfrynmuxs.png)

![The average successful proxy response time in October 2020 was as usual — 2.3 seconds](https://dev-to-uploads.s3.amazonaws.com/i/ahikilfl41xf95xnoq4j.png)

Looking further, parsing took about three seconds for searches with one hundred results (`num=100&tbm=shop`). HTML pages with one hundred Google Shopping results are 1.5 — 2MiB in size.

```bash
$ curl -s -A 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.122 Safari/537.36' 'https://www.google.com/search?q=lawn+mower&oq=lawn+mower&uule=w+CAIQICIdQXVzdGluLFRYLFRleGFzLFVuaXRlZCBTdGF0ZXM&hl=en&gl=us&num=500&tbm=shop&tbs=p_ord:rv&start=0&sourceid=chrome&ie=UTF-8' | wc -c | numfmt --to=iec-i --suffix=B
1.8MiB
```

### Profiling

We (naively) profiled our code and found out that image extraction via regexes from the inline JavaScript is slow but `Nokogiri::XML::XPathContext#evaluate` took a bit more.

```bash
$ TEST_RUBY_PROF=1 bundle exec rspec spec/search/google/shopping_results/shopping_results_card_results_spec.rb$ cat tmp/test_prof/ruby-prof-report-flat-wall-total.txt
Measure Mode: wall_time
Thread ID: 9714020
Total: 11.660184
Sort by: self_time

 %self      total      self      wait     child     calls  name                           location
 18.38      2.840     2.143     0.698     0.000       19   <Module::Kernel>#select
 13.88      1.626     1.619     0.005     0.002     2531   Nokogiri::XML::XPathContext#evaluate
  9.28      1.237     1.082     0.139     0.016      187   JSImageExtractor#extract_js_image js_image_extractor.rb:2
  4.22      0.566     0.492     0.000     0.074      488   <Module::Bootsnap::CompileCache::Native>#fetch
  3.97      0.463     0.463     0.000     0.000        5   <Class::Nokogiri::HTML::Document>#read_memory
  3.26      0.381     0.380     0.001     0.000    88163   String#split

* recursively called methods
```

Then we looked into flame graphs from `rbspy` and found out that `Nokogiri::HTML` takes 25% of wall time.

![`Nokogiri::HTML` takes 25% time of data extraction from HTML](https://dev-to-uploads.s3.amazonaws.com/i/azcf5loc5xu99kjkiz8m.png)

### Benchmarking Nokogiri with different versions of libxml2

Looked in `nokogiri` and found out that [before v1.11.0 it builds `libxml2` without compiler optimizations][9]. We’ve tried Nokogiri with [`--use-system-libraries`][10] and `Nokogiri::HTML` execution time decreased from 2.7 seconds to 1.5 seconds.

Timing results with system libraries

```bash
# Install system libxml2 and libxslt
$ gem install nokogiri --use-system-libraries$ bundle exec rails runner tmp/profiler.rbMeasure Mode: wall_time
Thread ID: 70292512504140
Fiber ID: 70292321749400
Total: 1.507682
Sort by: self_time%self      total      self      wait     child     calls  name                           location
 43.31      0.654     0.653     0.000     0.001     1128   Nokogiri::XML::XPathContext#evaluate
 13.84      0.211     0.209     0.002     0.000        1   <Class::Nokogiri::HTML::Document>#read_memory
  8.40      0.142     0.127     0.008     0.008      100   JSImageExtractor#extract_js_image js_image_extractor.rb:2
```

Regular nokogiri libraries

```bash
# Install system libxml2 and libxslt
$ gem install nokogiri --use-system-libraries$ bundle exec rails runner tmp/profiler.rbMeasure Mode: wall_time
Thread ID: 69919886342480
Fiber ID: 69919692333960
Total: 2.854719
Sort by: self_time%self      total      self      wait     child     calls  name                           location
 41.16      1.190     1.175     0.015     0.000        1   <Class::Nokogiri::HTML::Document>#read_memory
 31.82      0.910     0.908     0.000     0.001     1128   Nokogiri::XML::XPathContext#evaluate
  4.40      0.146     0.125     0.013     0.008      100   JSImageExtractor#extract_js_image js_image_extractor.rb:2
```

At that time I wasn’t sure how to properly profile Ruby code, so I’ve used everything: `rbspy`, `stackprof`, and `ruby-prof`.

```bash
$ cat tmp/profiler.rb
require "ruby-prof"profile = RubyProf.profile do
  search_params = {engine: "google", q: "roller blades", location: "Austin, United States", google_domain: "google.com", hl: "en", gl: "us", num: "500", device: "desktop", tbm: "shop", tbs: "p_ord:rv", file_path: "tmp/roller-blades.html"} Search.new(search_params).parse!
endprinter = RubyProf::FlatPrinter.new(profile)
printer.print($stdout, min_percent: 2)
```

### `Oga`

[Julien Khaleghy](https://twitter.com/hartator) also tried [`Oga` gem][12] instead of Nokogiri. It was about six times faster than Nokogiri.

```
Warming up --------------------------------------
            Nokogiri     1.000  i/100ms
                 Oga     1.000  i/100ms
Calculating -------------------------------------
            Nokogiri      0.913  (± 0.0%) i/s -     28.000  in  30.688232s
                 Oga      6.048  (±16.5%) i/s -    176.000  in  30.016426sComparison:
                 Oga:        6.0 i/s
            Nokogiri:        0.9 i/s - 6.62x  (± 0.00) slower
```

But some tests were failing with `LL::ParserError` from [`ruby-ll`][13] that is used in Oga.

```
$ bundle exec rspec specAn error occurred while loading ./spec/search/google/local_results/local_results_for_specific_place_spec.rb.
Failure/Error: if query_displayed = doc.at_css('input[name=q]')LL::ParserError:
  Unexpected T_IDENT for rule 24
# /Library/Ruby/Gems/2.6.0/gems/ruby-ll-2.1.2/lib/ll/driver.rb:15:in `parser_error'An error occurred while loading ./spec/search/google/sports_results/milwaukee_bucks_spec.rb.
Failure/Error: if query_displayed = doc.at_css('input[name=q]')LL::ParserError:
  Unexpected T_IDENT for rule 24
# /Library/Ruby/Gems/2.6.0/gems/ruby-ll-2.1.2/lib/ll/driver.rb:15:in `parser_error'An error occurred while loading ./spec/search/yahoo/organic_results/organic_results_coffee_spec.rb.
Failure/Error: if ad_result_node.classes.include? 'AdTop'NoMethodError:
  undefined method `classes' for #<Oga::XML::Element:0x00007fa7bbd8aa58>
  Did you mean?  classFinished in 0.00004 seconds (files took 6.79 seconds to load)
0 examples, 0 failures, 6 errors occurred outside of examples
```

We profiled our code once again. `at_css` took more time than `extract_js_image` for these shopping results. I guessed that was because we use multiple CSS selectors to support old and new Google layouts.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/1ojmio729o0xbxctfxip.png)

### Solution #1

I’ve compiled `nokogiri` with `-O2` compiler optimization and now its parsing performance was close to `oga`. I’ve added this workaround to [the Nokogiri issue on GitHub][14].

```
$ CFLAGS="-O2 -pipe" gem install nokogiri# Run benchmarkWarming up --------------------------------------
            Nokogiri     1.000  i/100ms
                 Oga     1.000  i/100ms
Calculating -------------------------------------
            Nokogiri      4.706  (± 0.0%) i/s -    140.000  in  30.039766s
                 Oga      5.585  (±17.9%) i/s -    166.000  in  30.030045sComparison:
                 Oga:        5.6 i/s
            Nokogiri:        4.7 i/s - same-ish: difference falls within error
```

It worked because `CFLAGS` are passed [here][15] and [there][16] in `ext/nokogiri/extconf.rb`.

We also tried different optimization levels. `-Ofast` was a bit faster than `-O2` and `-Ofast` for me. `-O2` and `-O3` are almost the same. 20ms speedup for`Nokogiri::HTML` not worth the risks of aggressive optimizations.

We reinstalled Nokogiri with `-O2` on production servers. In production, `-O2` gave about 600 ms (3.3 seconds vs 3.9 seconds) speed up for large search results pages.

### Pull request to Nokogiri

So we’ve opened a [PR to Nokogiri to compile libxml2 with `-O2 -g` flags][19]. After some discussion, [Mike Dalessio][20] came up with [heavy refactoring of _extconf.rb_][21] which also included changes from my PR. I have enjoyed communication with Mike.

## Images extraction from inline JavaScript with regular expressions

`extract_js_image` took 49% wall time of data extraction from Google Shopping HTML with `num=100` parameter. The entire parsing took two seconds.

```
$ bundle exec stackprof tmp/stackprof.dump --text --limit 20
==================================
  Mode: wall(1000)
  Samples: 986 (50.00% miss rate)
  GC: 18 (1.83%)
==================================
     TOTAL    (pct)     SAMPLES    (pct)     FRAME
       490  (49.7%)         467  (47.4%)     extract_js_image
       334  (33.9%)         334  (33.9%)     Nokogiri::XML::Document#decorate
        23   (2.3%)          23   (2.3%)     Nokogiri::XML::XPathContext#register_namespaces
        23   (2.3%)          23   (2.3%)     JSUtils.unescape
        15   (1.5%)          13   (1.3%)     Nokogiri::HTML::Document.parse
        15   (1.5%)          13   (1.3%)     Mongoid::Fields::ClassMethods#database_field_name
         9   (0.9%)           9   (0.9%)     (sweeping)
         8   (0.8%)           8   (0.8%)     (marking)
        13   (1.3%)           7   (0.7%)     #<Module:0x00007fb3c0edab68>.xpath_for
         7   (0.7%)           7   (0.7%)     block (3 levels) in class_attribute
        10   (1.0%)           6   (0.6%)     Nokogiri::XML::Searchable#extract_params
         4   (0.4%)           4   (0.4%)     Nokogiri::XML::Node#namespaces
       890  (90.3%)           4   (0.4%)     get_shopping_results
```

We found out with [Žilvinas Kučinskas][22], that `extract_js_image` matched the entire HTML string against seven regular expressions on each method call. `get_thumbnail` calls `extract_js_image`. `get_thumbnail` is called fifty times from different parsers. The parser for [Google Images API][23] called `get_thumbnail` one hundred times.

`extract_js_image` looked like this:

```ruby
if html =~ REGEX_1
  JSUtils.unescape $1
elsif html =~ REGEX_2
  JSUtils.unescape $1
elsif html =~ REGEX_3
  JSUtils.unescape $1
elsif html =~ REGEX_4
  JSUtils.unescape $1
elsif html =~ REGEX_5
  JSUtils.unescape $1
elsif html =~ REGEX_6
  JSUtils.unescape $1
elsif html =~ REGEX_7
  JSUtils.unescape $1
end
```

### Solution #2

The first assumption was to call `String#scan` once per parsing and cache extracted matches. We worked more on this assumption and came up with regular expressions with named captures.

```ruby
THUMBNAIL_CAPTURE_NAME = "thumbnail"
THUMBNAIL_ID_CAPTURE_NAME = "thumbnail_id"

THUMBNAIL_ID_REGEX = %r{(?<#{THUMBNAIL_ID_CAPTURE_NAME}>[dws_-',]{1,500})}

JS_IMAGE_REGEXES_GOOGLE = [
  %r{#{THUMBNAIL_ID_REGEX}CUT(?<#{THUMBNAIL_CAPTURE_NAME}>(?:CUT)[^"]+)}, # Omitted
]

JS_IMAGE_REGEXES_BING = [
  %r{#{THUMBNAIL_ID_REGEX}CUT(?<#THUMBNAIL_CAPTURE_NAME}>CUT[^<]+)}, # Omitted
]

JS_IMAGE_REGEXES = {
  google: JS_IMAGE_REGEXES_GOOGLE,
  bing: JS_IMAGE_REGEXES_BING,

  all: JS_IMAGE_REGEXES_GOOGLE + JS_IMAGE_REGEXES_BING
}
```

We used code from [this StackOverflow answer][24] to convert all named captures from the `String#scan` to the single dictionary of `{ “thumbnail_id” => “thumbnail” }`.

```ruby
def extracted_thumbnails
  return @extracted_thumbnails if @extracted_thumbnails.present?

  js_image_regexes = JS_IMAGE_REGEXES.detect { |key, _| engine.starts_with?(key.to_s) }&.last || JS_IMAGE_REGEXES[:all]@extracted_thumbnails = js_image_regexes.collect { |regex|
    regex_capture_names = regex.namesthumbnail_index = regex_capture_names.index(THUMBNAIL_CAPTURE_NAME)
    thumbnail_id_index = regex_capture_names.index(THUMBNAIL_ID_CAPTURE_NAME)

    html.scan(regex).collect do |match|
      found_thumbnail = match[thumbnail_index]
      found_thumbnail_id = match[thumbnail_id_index]

      found_thumbnail_id.split(",").map { |thumb| Hash[thumb.tr("'", "").squish, found_thumbnail] }
    end
  }.flatten.inject(:merge) || {}
end
```

`extracted_thumbnails` allocates about 2.25 MB of memory for large HTML like Google Shopping results. That’s 0.2 MB more comparing to the previous slow implementation.

Usage of `extracted_thumbnails` is straightforward.

```ruby
def extract_js_image(image_node)
  return unless image_node thumbnail_id = (image_node["id"])

  return unless thumbnail_id if (found_thumbnail = extracted_thumbnails[thumbnail_id])
    JSUtils.unescape(found_thumbnail)
  end
end
```

`extract_js_image` takes 17.9% wall time of data extraction from Google shopping HTML with `num=100` parameter. The entire parsing takes 1.2 seconds.

```bash
$ bundle exec stackprof tmp/stackprof.dump --text --limit 20
==================================
  Mode: wall(1000)
  Samples: 532 (58.18% miss rate)
  GC: 15 (2.82%)
==================================
     TOTAL    (pct)     SAMPLES    (pct)     FRAME
       253  (47.6%)         253  (47.6%)     Nokogiri::XML::Document#decorate
       107  (20.1%)          95  (17.9%)     extract_js_image
        23   (4.3%)          23   (4.3%)     Nokogiri::XML::XPathContext#register_namespaces
        12   (2.3%)          10   (1.9%)     JSUtils.unescape
        11   (2.1%)           9   (1.7%)     Nokogiri::HTML::Document.parse
```

Nokogiri#at_css
================

Then we ran `rbspy` in production and found out that `at_css` (`Nokogiri::XML::Document#decorate`) took much time because of [performance problem in `libxml2`][26] (the underlying lib that `nokogiri` uses to parse and traverse XML).

![The flame graph shows that there’s something wrong with the C function. Interesting…](https://dev-to-uploads.s3.amazonaws.com/i/xxumen54vjp9g31icrao.png)

`c function` is [not very helpful][27] to find the performance problem, so we dug deeper.

perf Linux profiler
-------------------

I searched over the web how to profile C extensions for Ruby and C code in general, and found out [Brendan Gregg’s tutorial on Linux perf][28]. That was my first usage of [Linux `perf` profiler][31]. I’ve also tried [`gperftools`][32] and [`pprof`][33], because seen [its usage][34]. And [`flamescope`][35], because it was made by Brendan Gregg. There are many similar tools and it was hard to figure out what to use during two weeks or so.

I reinstalled Nokogiri with debugging info, not-stripped just in case.

```bash
CFLAGS="-O2 -ggdb3 -gdwarf -pipe -lprofiler -fno-omit-frame-pointer" gem install nokogiri
```

Executing `perf record` and `perf report` shows that`xmlXPathCompOpEval` and `xmlXPathNodeCollectAndTest` took most of the time, and are being called recursively.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/x4iq02j6judbewr2ihjq.png)

The flame graph for the single `search.parse!` shown basically the same.

```bash
cargo install flamegraph
flamegraph -- bundle exec rails runner 'Search.new(q: "pc game", tbm: "shop", file_path: "tmp/pc_game_big_shopping.html").parse!;
```

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/xm60x3hbabwuwzjy34vl.png)

Chart from [`pprof`][36] shows that `objspace_malloc_increase.constprop.0`, `atomic_sub_nounderflow` and `xmlStrlen` have the biggest total and self time. `xmlXPathNodeCollectAndTest` has the biggest self time which means the body of this function is a potential performance bottleneck.

I’ve installed [`perf_data_converter`][37] to be able to use `perf.data` report with `pprof`.

```bash
perf record -F99 -e cycles:u -g -- bundle exec rails runner 'Search.new(q: "pc game", tbm: "shop", file_path: "tmp/pc_game_big_shopping.html").parse!;'

pprof -web perf.data
```

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/6iciy6r1dco38z47l5hd.png)

[`flamescope`][38] shows the same as [`flamegraph`][39]. Both of these tools use the same tools to generate chart probably.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/chsslc7phupr2vlkxh2o.png)

Returning back to Nokogiri
--------------------------

I’ve opened an [issue in the Nokogiri repository][40]. I had ideas for improving `xmlStrlen` performance but it was better to get help from someone who knows C and has experience with `libxml2`.

In the big document, `xmlStrlen` is called 15K times for the same string. In the small document, it’s being called for 5K times. Three times bigger document and three times more calls. But the strings are the same. My assumption as a person who doesn’t know C: the unique string pointer address means that the function was called with the same argument.

Given this Ruby script that searches an element in the container.

```ruby
html = File.read(ARGV[0])

doc = Nokogiri::HTML.parse(html)

10.times do
  doc.css(".sh-dlr__list-result, .sh-dgr__grid-result").each do |sh_r|
    10.times do
      sh_r.at_css(".na4ICd:not(.rOwove):not(.TxCHDf):nth-of-type(1), .hBUZL:not(.Rv2Cae):nth-of-type(2), .hBUZL:not(.Rv2Cae):not(.Fxxvzc):not(:has(span)):not(:has(div)), .dWRflb, .p7n7Ze:not(:has(a))")
    end
  end
end

```

I’ve calculated `xmlStrlen` calls by attaching dynamic breakpoint in `gdb`. To do so, I recompiled nokogiri with debug information.

```bash
CFLAGS="-O -ggdb3 -pipe -fno-omit-frame-pointer" gem install nokogiri
```

Then attached the breakpoint via gdb to get the line to attach `dprintf` later on.

```bash
bundle exec gdb -q -ex 'set breakpoint pending on' --ex 'thread apply all bt full' --ex 'b xmlStrlen' --ex run --args ruby ./tmp/slow_search_parse.rb tmp/pc_game_big_shopping.html
```

Then attached `[dprintf][41]` via GDB to the `xmlStrlen` function of `libxml2` to print pointer of `str` variable and log output to the file. I executed this script with a big and small HTML document.

```bash
bundle exec gdb -q -ex 'set breakpoint pending on' --ex 'thread apply all bt full' --ex 'dprintf xmlstring.c:425, "str: %pn", str' --ex run --ex quit --args ruby ./tmp/slow_search_parse.rb tmp/pc_game_big_shopping.html |& tee xmlStrlen_big.log

bundle exec gdb -q -ex 'set breakpoint pending on' --ex 'thread apply all bt full' --ex 'dprintf xmlstring.c:425, "str: %pn", str' --ex run --ex quit --args ruby ./tmp/slow_search_parse.rb tmp/pc_game_small_shopping.html |& tee xmlStrlen_small.log
```

Then examined logs

```bash
$ grep str: xmlStrlen_big.log | sort | uniq -c | sort -rn | head -20
  15446 str: 0x112e760
  15405 str: 0x112e6d0
  13405 str: 0x15ddb80
   4006 str: 0x7fffe6c615ad
   1962 str: 0x112e740
    381 str: 0x7fffe69de010
    374 str: 0x15b5520
    321 str: 0x1649d60
    306 str: 0x174c7a0
    306 str: 0x1734430
    306 str: 0x15db410
    306 str: 0x15b63b0
    303 str: 0x174c830
    303 str: 0x15db4a0
    303 str: 0x15b64a0
    298 str: 0x161a8b0
    297 str: 0x16e3fa0
    291 str: 0x160fd50
    287 str: 0x163c320
    285 str: 0x16feef0

$ grep str: xmlStrlen_small.log | sort | uniq -c | sort -rn | head -20
   5046 str: 0xf3fbc0
   5038 str: 0xf3fcb0
   1687 str: 0xf40440
    906 str: 0xf3fc30
    806 str: 0x7fffe6c615ad
    431 str: 0xfe0460
    420 str: 0x11b53e0
    384 str: 0x1067bb0
    353 str: 0xfdc760
    328 str: 0x11e5ab0
    322 str: 0x11e5a40
    306 str: 0x11ccea0
    303 str: 0x11ccf30
    303 str: 0x11bc040
    301 str: 0x11a57d0
    291 str: 0x11e7740
    289 str: 0x1200a70
    288 str: 0x11d5660
    284 str: 0x11e6d80
    284 str: 0x11c3fd0
```

I had a simple idea about caching the length of strings in memory in `xmlStrlen` and compare the performance, but [Mike Dalessio][42] said [it’s not safe to cache string lengths across `xmlStrlen` calls][45].

Then Mike came up with a [2x speedup of node lookups][46] which decreased the time to extract data from big HTML files to one second.

Improvement of xmlStrlen from libxml2
=====================================

We’ve moved further with an idea to cache string lengths in `xmlStrlen` and compared its performance with `strlen` from `glibc`.

Given the simplest benchmark, `xmlStrlen` is two times slower than [`strlen` from `glibc`][49] on small strings, ten times slower on average strings, and thirty times slower on big strings and an entire HTML file.

The performance of `xmlStrlen` resulted in the 0.7 - 1.5 seconds to parse and search through 2 MB HTML document by using Nokogiri. I described this issue [in the Nokogiri repository][50] which led to [2x speedup on the Nokogiri side][51], but `xmlStrlen` still could be faster.

```bash
$ ./slow_parsing_benchmark
xmlStrlen (entire HTML file): 926171.936981 μs
glibc_xmlStrlen (entire HTML file): 36905.903992 μs
delta (xmlStrlen ÷ glibc_xmlStrlen): 25.094584 timesxmlStrlen (average string): 57479.204010 μs
glibc_xmlStrlen (average string): 5802.069000 μs
delta (xmlStrlen ÷ glibc_xmlStrlen): 9.905937 timesxmlStrlen (bigger string): 388056.315979 μs
glibc_xmlStrlen (bigger string): 12797.856995 μs
delta (xmlStrlen ÷ glibc_xmlStrlen): 30.318382 timesxmlStrlen (smallest string): 15870.046021 μs
glibc_xmlStrlen (smallest string): 6282.208984 μs
delta (xmlStrlen ÷ glibc_xmlStrlen): 2.527903 times
```

So I’ve opened an [issue in `libxml2`][53]. The naive approach to simply reuse `strlen` in `xmlStrlen` speed up our document parsing and searching from 1.4 seconds to about 800 ms.

```diff
diff --git a/xmlstring.c b/xmlstring.c
index e8a1e45d..df247dff 100644
--- a/xmlstring.c
+++ b/xmlstring.c
@@ -423,14 +423,9 @@ xmlStrsub(const xmlChar *str, int start, int len) {

 int
 xmlStrlen(const xmlChar *str) {
-    int len = 0;
-
     if (str == NULL) return(0);
-    while (*str != 0) { /* non input consuming */
-        str++;
-        len++;
-    }
-    return(len);
+
+    return strlen((const char*)str);
 }

 /**
```

[Mike Dalessio][54] noted that `xmlStrlen()` has remained unchanged since the commit it was introduced, [`260a68fd`][55], in 1998, and is equivalent to the K&R version.

glibc’s implementation is faster because it’s implemented in assembly customized for common/modern CPUs. Some background [here][56].

With some help, I’ve posted a [very verbose comment][57] of 10% speedup with the `xmlStrlen` patch on the C program that used libxml2 directly.

Before the patch

```bash
$ cd ~/code/libxml2
$ git checkout origin/master
$ CFLAGS="-O2 -pipe -g" ../configure --host=x86_64-pc-linux-gnu --enable-static --disable-shared --with-iconv=yes --without-python --without-readline --with-c14n --with-debug --with-threads && make clean && make -j
$ sudo make install
$ cd ~/code/benchmark && make && ./slow_parsing_benchmark ./pc_game_big_shopping.html "//*[contains(concat(' ', @class, ' '), ' sh-dlr__list-result ')]//*[contains(concat(' ', @class, ' '), ' hBUZL ')]"
# Some parsing errors like
./pc_game_big_shopping.html:62: HTML parser error : Tag path invalid
4h2c0-1.1.9-2 2-2s2 .9 2 2c0 2-3 1.75-3 5h2c0-2.25 3-2.5 3-5 0-2.21-1.79-4-4-4z"searching './pc_game_big_shopping.html' with '//*[contains(concat(' ',@class,' '), ' sh-dlr__list-result ')]//*[contains(concat(' ', @class, ' '), ' hBUZL ')]' 1000 times
NODESET with 260 results
67855 ms
```

After the patch

```bash
$ cd ~/code/libxml2
$ git checkout xmlStrlen-patch
$ CFLAGS="-O2 -pipe -g" ../configure --host=x86_64-pc-linux-gnu --enable-static --disable-shared --with-iconv=yes --without-python --without-readline --with-c14n --with-debug --with-threads && make clean && make -j
$ sudo make install
$ cd ~/code/benchmark && make && ./slow_parsing_benchmark ./pc_game_big_shopping.html "//*[contains(concat(' ',@class,' '), ' sh-dlr__list-result ')]//*[contains(concat(' ', @class, ' '), ' hBUZL ')]"
searching './pc_game_big_shopping.html' with '//*[contains(concat(' ',@class,' '), ' sh-dlr__list-result ')]//*[contains(concat(' ', @class, ' '), ' hBUZL ')]' 1000 times
NODESET with 260 results
59767 ms
```

I was not sure why it ran faster with the patch. The flame graph of the sample program from [this post][66] looked the same for nokogiri on `master` and on the branch of this PR. [`xmlXPathCompOpEval`][67] is dominating before and after the patch.

![In both cases, `xmlXPathNodeCollectAndTest` and `xmlXPathNodeSetMergeAndClear` are taking most of the time of `xmlXPathCompOpEval`](https://dev-to-uploads.s3.amazonaws.com/i/kpd3glstqx3692w9iqmw.png)

[Nick Wellnhofer][68] hasn’t responded to my comment and in [the PR to libxml2][69], but [our PR to Nokogiri was merged to master][70] and shipped in v1.11.0.rc4, so the solution was good enough for us.

*Update February 21, 2022: [Mike's PR with the `xmlStrlen` optimization](https://gitlab.gnome.org/GNOME/libxml2/-/mergerequests/147/diffs) was merged to Libxml2.*

Results
=======

Data extraction from big HTML files decreased from three seconds to one second and two of our PRs were shipped in [Nokogiri v1.11.0][71].

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/5e2t909cbo35w7qkd3mh.png)

Two of our PRs were shipped to [Nokogiri v1.11.0][71]

Things we haven’t tried
=======================

1. Haven’t tried to use [bcc for tracing and profiling][73].
2. Haven’t read the entire [documentation about `perf`][75].

What’s next
===========

I’m glad to have the opportunity to contribute to an open-source project that is used by thousands of people. Hopefully, we will speed up Nokogiri (or XML parser it uses) to match the performance of [`html5ever`][76] or [`lexbor`][77] at some point in the future. 800 ms to extract data from HTML is still too much.

As of an experiment, I’ve made [an FFI wrapper around the Rust `scraper` crate][80]. `at_css.text` calls of proof of concept are 60 times faster than Nokogiri ones.

```bash
$ ruby benchmarks/nokogiri_benchmark.rb
Warming up --------------------------------------
      Nokogiri::HTML     2.000  i/100ms
  NokogiriRust::HTML     3.000  i/100ms
Calculating -------------------------------------
      Nokogiri::HTML     27.195  (±14.7%) i/s -    132.000  in   5.042868s
  NokogiriRust::HTML     40.319  (± 5.0%) i/s -    201.000  in   5.001218sComparison:
  NokogiriRust::HTML:       40.3 i/s
      Nokogiri::HTML:       27.2 i/s - 1.48x  (± 0.00) slowerWarming up --------------------------------------
Nokogiri::HTML.at_css.text
                         5.000  i/100ms
NokogiriRust::HTML.at_css.text
                       394.000  i/100ms
Calculating -------------------------------------
Nokogiri::HTML.at_css.text
                         61.027  (± 3.3%) i/s -    305.000  in   5.002827s
NokogiriRust::HTML.at_css.text
                          3.900k (± 2.9%) i/s -     19.700k in   5.056373sComparison:
NokogiriRust::HTML.at_css.text:     3899.6 i/s
Nokogiri::HTML.at_css.text:       61.0 i/s - 63.90x  (± 0.00) slower
```

No plans at the moment, but I think about making an adapter between `html5ever` and Nokogiri in a similar way to [Nokogumbo][81]. In this case, the project may become more long-term and even be used as the main HTML parsing library for Nokogiri like [gumbo is going to become in 2021][82].

[4]: https://serpapi.canny.io/bugs/p/inflated-actual-response-times-from-html-to-json-parsing
[5]: https://serpapi.com/shopping-results
[9]: https://github.com/sparklemotion/nokogiri/issues/2020#issuecomment-615948385
[10]: https://nokogiri.org/tutorials/installing_nokogiri.html#install-with-system-libraries
[12]: https://gitlab.com/yorickpeterse/oga
[13]: https://gitlab.com/yorickpeterse/ruby-ll
[14]: https://github.com/sparklemotion/nokogiri/issues/2020#issuecomment-707110243
[15]: https://github.com/sparklemotion/nokogiri/blob/793baaee7c32b2bfefc619c4c41c378c53b68c3f/ext/nokogiri/extconf.rb#L529
[16]: https://github.com/sparklemotion/nokogiri/blob/793baaee7c32b2bfefc619c4c41c378c53b68c3f/ext/nokogiri/extconf.rb#L297
[19]: https://github.com/sparklemotion/nokogiri/pull/2100
[20]: https://twitter.com/flavorjones
[21]: https://github.com/sparklemotion/nokogiri/pull/2100#issuecomment-716750059
[22]: https://twitter.com/zilkucinskas
[23]: https://serpapi.com/images-results
[24]: https://stackoverflow.com/a/9485453/1291371
[26]: https://github.com/sparklemotion/nokogiri/issues/1861
[27]: https://github.com/rbspy/rbspy/issues/110
[28]: http://www.brendangregg.com/perf.html
[31]: https://github.com/torvalds/linux/tree/76c057c84d286140c6c416c3b4ba832cd1d8984e/tools/perf
[32]: https://github.com/gperftools/gperftools
[33]: https://github.com/google/pprof
[34]: https://github.com/sparklemotion/nokogiri/issues/1861#issuecomment-454650144
[35]: https://github.com/Netflix/flamescope/
[36]: https://github.com/google/pprof
[37]: https://github.com/google/perf_data_converter
[38]: https://github.com/Netflix/flamescope/
[39]: https://github.com/flamegraph-rs/flamegraph
[40]: https://github.com/sparklemotion/nokogiri/issues/2133
[41]: https://doc.ecoscentric.com/gnutools/doc/gdb/Dynamic-Printf.html
[42]: https://twitter.com/flavorjones
[45]: https://github.com/sparklemotion/nokogiri/issues/2133#issuecomment-747665909
[46]: https://github.com/sparklemotion/nokogiri/issues/2133#issuecomment-748280589
[49]: https://sourceware.org/git/?p=glibc.git;a=blob;f=string/strlen.c;h=f934e1a1e897c0176c04566a182f092dc1ab2a37;hb=HEAD
[50]: https://github.com/sparklemotion/nokogiri/issues/2133
[51]: https://github.com/sparklemotion/nokogiri/pull/2137
[53]: https://gitlab.gnome.org/GNOME/libxml2/-/issues/212
[54]: https://twitter.com/flavorjones
[55]: https://gitlab.gnome.org/GNOME/libxml2/-/commit/260a68fd34302f352aa8f4c2f2901cefa3e1d2f7
[56]: https://stackoverflow.com/questions/57650895/why-does-glibcs-strlen-need-to-be-so-complicated-to-run-quickly/57676035#57676035
[57]: https://gitlab.gnome.org/GNOME/libxml2/-/issues/212#note_992091
[66]: https://gitlab.gnome.org/GNOME/libxml2/-/issues/212#note_991383
[67]: https://gitlab.gnome.org/GNOME/libxml2/blob/e6ec58ecf7adf73335cfb40f0d5bc673681f766b/xpath.c#L12960
[68]: https://gitlab.gnome.org/nwellnhof
[69]: https://gitlab.gnome.org/GNOME/libxml2/-/merge_requests/95
[70]: https://github.com/sparklemotion/nokogiri/pull/2144#issuecomment-751844468
[71]: https://github.com/sparklemotion/nokogiri/releases/tag/v1.11.0
[73]: https://github.com/iovisor/bcc/blob/master/tools/lib/ucalls_example.txt
[75]: https://github.com/torvalds/linux/tree/33dc9614dc208291d0c4bcdeb5d30d481dcd2c4c/tools/perf/Documentation
[76]: https://github.com/servo/html5ever
[77]: https://github.com/lexbor/lexbor
[80]: https://github.com/ilyazub/nokogiri-rust#nokogirirust
[81]: https://github.com/rubys/nokogumbo/blob/b3a8e5e5a0fbd1502fc0b08eaeb3af6cc490c88c/lib/nokogumbo/html5.rb#L7-L9
[82]: https://github.com/sparklemotion/nokogiri/issues/2064#issuecomment-685007225