How to protect content against scrapers with headless browsers?
Tags: scrapers, bot, webdev, programming
Published: 2022-11-01 20:23:11

<!-- wp:paragraph -->
<p>We are living in a century of information. The data could bring huge value to your business. That's why a lot of different scrapers exist in the world. I guess each owner of a website at least once was thinking about how to protect their content. One of the ways is to use JavaScript to render content. However, modern technologies allow running a headless browser on the server and scraping all required data. Let's make scraper's creator life a bit harder…</p>
<!-- /wp:paragraph -->

<!-- wp:more -->
<!--more-->
<!-- /wp:more -->

<!-- wp:paragraph -->
<p>I want to show you an example of how to protect content from scrapers that use headless browsers for rendering content. However, first, I need to tell you some theories. Feel free to skip the next paragraph if you are a smart ass. </p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>Shadow Root</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Let me introduce you <strong>Shadow Root</strong>. If one of you doesn't know what it is, I'll try to explain. Shadow Root allows you to create separate DOM inside your current DOM. It's kind of a document inside a document. Why could it be useful? For example, you can make your web component with your HTML IDs and CSS classes without taking care of other code on the web page. If you want to learn more, try to read <a href="https://developer.mozilla.org/en-US/docs/Web/API/ShadowRoot" target="_blank" rel="noreferrer noopener nofollow">documentation</a> and this <a href="https://javascript.info/shadow-dom" target="_blank" rel="noreferrer noopener nofollow">article</a>. </p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>Protection</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>So, how can we use shadow root to protect content from scrapers? Shadow Root has two modes: open and closed. If the mode is closed, then you can't access elements inside JavaScript. That makes it impossible to scrape content from those elements by evaluating JS inside the browser.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>I'll show you an example. Let's create an empty HTML document with a div for sensitive data. </p>
<!-- /wp:paragraph -->

<!-- wp:enlighter/codeblock -->
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">&lt;strong>Following content should be protected:&lt;/strong>
&lt;div id="protected-content">
&lt;/div></pre>
<!-- /wp:enlighter/codeblock -->

<!-- wp:paragraph -->
<p>We need to use JavaScript, so let's create a script block after our div and put the following code for the test. I use shadow root in an open mode for now.</p>
<!-- /wp:paragraph -->

<!-- wp:enlighter/codeblock -->
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">&lt;strong>Following content should be protected:&lt;/strong>
&lt;div id="protected-content">
&lt;/div>
&lt;script>
    const element = document.querySelector('#sensitive-content');
    element.attachShadow({ mode: 'open' });
    element.shadowRoot.innerHTML = '&lt;p>Hello from protected area, buddy!&lt;/p>';
&lt;/script></pre>
<!-- /wp:enlighter/codeblock -->

<!-- wp:paragraph -->
<p>What did we get as a result?</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":2555,"sizeSlug":"large","linkDestination":"none"} -->
<figure class="wp-block-image size-large"><img src="https://jakeroid.com/wp-content/uploads/2022/04/image-7-1024x530.png" alt="Shadow root example in open mode" class="wp-image-2555"/><figcaption>Shadow root example in open mode</figcaption></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Looks like our shadow root code is working. However, it didn't give any protection. You could easily access it by using the code like <em>element.shadowRoot</em>.</p>
<!-- /wp:paragraph -->

<!-- wp:enlighter/codeblock -->
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">element.attachShadow({ mode: 'closed' });</pre>
<!-- /wp:enlighter/codeblock -->

<!-- wp:paragraph -->
<p>Oops... Looks like my code doesn't work now. </p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":2543,"sizeSlug":"large","linkDestination":"none"} -->
<figure class="wp-block-image size-large"><img src="https://jakeroid.com/wp-content/uploads/2022/04/image-1-1024x104.png" alt="Shadow root is not accessible if mode is closed." class="wp-image-2543"/><figcaption>Shadow root is not accessible if mode is closed</figcaption></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>As I said before, closed shadow root can't be accessible from JavaScript. That means element.shadowRoot will always be null if the mode is closed. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>How can we fix that? Let's create a custom HTML component. At first, I removed div with id protected-content and the old JavaScript tag. Then I made a new script block inside the head and put the following code. The new code contains a declaration of the ProtectedWebComponent, the constructor, and connectedCallback method. You can find similar examples on the internet. </p>
<!-- /wp:paragraph -->

<!-- wp:enlighter/codeblock -->
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">class ProtectedWebComponent extends HTMLElement {
    constructor() {
        super();
        this._protected_root = this.attachShadow({ mode: "closed" });
    }
    connectedCallback() {
        this._protected_root.innerHTML = `&lt;p>Hello from protected area, buddy!&lt;/p>`;
    }
}
window.customElements.define("protected-web-component", ProtectedWebComponent);</pre>
<!-- /wp:enlighter/codeblock -->

<!-- wp:paragraph -->
<p>Then I added a new component to HTML.</p>
<!-- /wp:paragraph -->

<!-- wp:enlighter/codeblock -->
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">&lt;strong>Following content should be protected:&lt;/strong>
&lt;protected-web-component>&lt;/protected-web-component></pre>
<!-- /wp:enlighter/codeblock -->

<!-- wp:paragraph -->
<p>Does it work? Yeah, looks good. </p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":2557,"sizeSlug":"large","linkDestination":"none"} -->
<figure class="wp-block-image size-large"><img src="https://jakeroid.com/wp-content/uploads/2022/04/image-8-1024x413.png" alt="Closed shadow root example. Scrapers can't access it." class="wp-image-2557"/><figcaption>Closed shadow root example</figcaption></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>However, there is still a problem. You can't access shadow root by calling something like <em>element.shadowRoot</em>, but there is still <em>_protected_root</em> field. So, it allows us to get shadow root by calling <em>element._protected_root</em>. How could we fix that? Let's initialize content inside the constructor and don't save the shadow root link anywhere. </p>
<!-- /wp:paragraph -->

<!-- wp:enlighter/codeblock -->
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">class ProtectedWebComponent extends HTMLElement {
    constructor() {
        super();
        const shadowRootLink = this.attachShadow({ mode: "closed" });
        shadowRootLink.innerHTML = `&lt;p>Hello from protected area, buddy!&lt;/p>`;
    }
}
window.customElements.define("protected-web-component", ProtectedWebComponent);</pre>
<!-- /wp:enlighter/codeblock -->

<!-- wp:paragraph -->
<p>Looks much better now.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":2553,"sizeSlug":"full","linkDestination":"none"} -->
<figure class="wp-block-image size-full"><img src="https://jakeroid.com/wp-content/uploads/2022/04/image-6.png" alt="protected content against scrapers" class="wp-image-2553"/><figcaption>It's impossible to access closed shadow root from JavaScript</figcaption></figure>
<!-- /wp:image -->

<!-- wp:heading -->
<h2>Source code</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>If you want to check the source code, welcome to my GitHub <a href="https://github.com/Jakeroid/protection-scraping-headless-browser" target="_blank" rel="noreferrer noopener">repository</a>. </p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>Results</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>As you can see, it's pretty to protect some content using shadow root. However, you should understand this way is not a magic stick. Using shadow root to protect content could make a web scraper developer's life more complex, but it doesn't put content in some bulletproof armor. If you are interested in how to avoid this protection, write a comment below, and I'll try to explain. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>By the way, I have an article <a href="https://jakeroid.com/blog/how-to-download-vimeo-video-using-python/">how to download Vimeo video using Python</a>. Maybe you are interested.</p>
<!-- /wp:paragraph -->