Zyte Proxy: Smart rotating proxy for web scraping
Tags: zyte, scraping, scrapinghub, scrapy
Published: 2021-03-13 02:35:58

<h2 id="496f" class="hk hl ft av hm hn ho hp hq hr hs ht hu hv hw hx hy hz ia ib ic id ie if ig ih ec">Struggling with managing your proxies when Web Scraping? Try Zyte! The Zyte by developed by&nbsp;<a class="bq ii" href="https://scrapinghub.com/?rfsn=3883267.be32c0" rel="noopener nofollow">Scrapinghub.com</a></h2>
<blockquote class="ij ik il">
<p id="eca7" class="im in io ip b iq ir hp is it iu ht iv iw ix iy iz ja jb jc jd je jf jg jh ji dc ec" data-selectable-paragraph=""><strong class="ip cp"><em class="ft">Let&rsquo;s face it, managing&nbsp;</em></strong><a class="bq ii" href="https://www.privateproxyreviews.com/residential-proxies/" rel="noopener nofollow"><strong class="ip cp"><em class="ft">your proxy pool</em></strong></a><strong class="ip cp"><em class="ft">&nbsp;is an absolute pain!&nbsp;</em></strong><em class="ft">Nothing annoys developers more than crawlers failing because their proxies are continuously getting banned.</em></p>
</blockquote>
<p id="d8fa" class="im in ft ip b iq ir hp is it iu ht iv iw ix iy iz ja jb jc jd je jf jg jh ji dc ec" data-selectable-paragraph="">Not only do you constantly find yourself firefighting proxy fires, the people who rely on this web data just get increasingly frustrated with you because of the unreliability of the data feed.</p>
<p id="47af" class="im in ft ip b iq ir hp is it iu ht iv iw ix iy iz ja jb jc jd je jf jg jh ji dc ec" data-selectable-paragraph="">We were in the same boat for years, until we hit our breaking point and decided to solve this problem forever.</p>
<p id="1ae2" class="im in ft ip b iq ir hp is it iu ht iv iw ix iy iz ja jb jc jd je jf jg jh ji dc ec" data-selectable-paragraph="">At the time, Scrapinghub was about 3 years in business, providing web scraping consultancy services to companies looking to outsource their data extraction.</p>
<p id="4192" class="im in ft ip b iq ir hp is it iu ht iv iw ix iy iz ja jb jc jd je jf jg jh ji dc ec" data-selectable-paragraph="">Then along came this one project&hellip;</p>
<blockquote class="ij ik il">
<p id="8aae" class="im in io ip b iq ir hp is it iu ht iv iw ix iy iz ja jb jc jd je jf jg jh ji dc ec" data-selectable-paragraph=""><em class="ft">The client wanted us to build a web scraping infrastructure to&nbsp;</em><a class="bq ii" href="https://medium.com/@jesaltnl/how-to-scrape-amazon-reviews-with-python-code-5fd8ab62d165" rel="noopener"><strong class="ip cp"><em class="ft">scape product data</em></strong></a><strong class="ip cp"><em class="ft">&nbsp;</em></strong><em class="ft">from 20 e-commerce sites, about 1 million requests per day. Which at the time was a big deal!.</em></p>
</blockquote>
<p id="9d52" class="im in ft ip b iq ir hp is it iu ht iv iw ix iy iz ja jb jc jd je jf jg jh ji dc ec" data-selectable-paragraph="">Everything starte<span id="rmm">d</span>&nbsp;off great. We developed the spiders, done a number of pilot crawls and delivered the data to the customer.</p>
<p id="cf67" class="im in ft ip b iq ir hp is it iu ht iv iw ix iy iz ja jb jc jd je jf jg jh ji dc ec" data-selectable-paragraph="">However, we ran into serious problems scaling the crawls.</p>
<p id="ce6a" class="im in ft ip b iq ir hp is it iu ht iv iw ix iy iz ja jb jc jd je jf jg jh ji dc ec" data-selectable-paragraph="">Although our spiders were well designed and configured to crawl at a polite speed, when we moved the project from proof of concept to production our proxies we being banned at an alarming rate.</p>
<p id="200b" class="im in ft ip b iq ir hp is it iu ht iv iw ix iy iz ja jb jc jd je jf jg jh ji dc ec" data-selectable-paragraph="">Eventually, it got to the point that we couldn&rsquo;t scale the crawl anymore as we couldn&rsquo;t put out the proxy fires fast enough.</p>
<p id="e37b" class="im in ft ip b iq ir hp is it iu ht iv iw ix iy iz ja jb jc jd je jf jg jh ji dc ec" data-selectable-paragraph="">Initially, we told the client that we&rsquo;d have the issue fixed in 1 or 2 days &ldquo;as it was just a matter of swapping out the banned IPs&rdquo;.</p>
<p id="40cd" class="im in ft ip b iq ir hp is it iu ht iv iw ix iy iz ja jb jc jd je jf jg jh ji dc ec" data-selectable-paragraph="">However, the days kept ticking by and we still hadn&rsquo;t found a permanent solution.</p>
<p id="a7ad" class="im in ft ip b iq ir hp is it iu ht iv iw ix iy iz ja jb jc jd je jf jg jh ji dc ec" data-selectable-paragraph="">Finally, nearly a month later. We fixed it!</p>
<p id="4745" class="im in ft ip b iq ir hp is it iu ht iv iw ix iy iz ja jb jc jd je jf jg jh ji dc ec" data-selectable-paragraph="">The solution&hellip;</p>
<p id="ac00" class="im in ft ip b iq ir hp is it iu ht iv iw ix iy iz ja jb jc jd je jf jg jh ji dc ec" data-selectable-paragraph=""><strong class="ip cp">We stopped focusing on the underlying IPs and put all our energy into intelligently managing the IPs so that we could&nbsp;</strong><a class="bq ii" href="https://www.privateproxyreviews.com/avoid-ip-ban-scraping-never-blocked-blacklisted/" rel="noopener nofollow"><strong class="ip cp">scrape reliably without the fear of being banned</strong></a><strong class="ip cp">.</strong></p>
<p id="47c7" class="im in ft ip b iq ir hp is it iu ht iv iw ix iy iz ja jb jc jd je jf jg jh ji dc ec" data-selectable-paragraph="">This breakthrough was a game-changer for us. With this new proxy management layer, we were able to scale our crawls nearly 100X and completely remove the headache of managing proxies.</p>
<p id="0e14" class="im in ft ip b iq ir hp is it iu ht iv iw ix iy iz ja jb jc jd je jf jg jh ji dc ec" data-selectable-paragraph="">This new proxy management layer would automatically select the&nbsp;<a class="bq ii" href="https://www.privateproxyreviews.com/" rel="noopener nofollow"><strong class="ip cp">best proxy to use for the target website</strong></a>&nbsp;and manage all the proxy rotation, throttling, blacklist, etc. ensuring that we could reliably extract the data we need.</p>
<p id="7842" class="im in ft ip b iq ir hp is it iu ht iv iw ix iy iz ja jb jc jd je jf jg jh ji dc ec" data-selectable-paragraph="">All without any manual intervention from our engineers!</p>
<p id="2cdb" class="im in ft ip b iq ir hp is it iu ht iv iw ix iy iz ja jb jc jd je jf jg jh ji dc ec" data-selectable-paragraph="">As we continued to scale, our customers increasingly were asking us how were we achieving such reliability with our proxies.</p>
<p id="34f1" class="im in ft ip b iq ir hp is it iu ht iv iw ix iy iz ja jb jc jd je jf jg jh ji dc ec" data-selectable-paragraph="">So in 2012, we decided to make this technology available to everyone in the form of&nbsp;<a class="bq ii" href="https://scrapinghub.com/crawlera?rfsn=3883267.be32c0" rel="noopener nofollow"><strong class="ip cp">Crawlera</strong></a>.</p>
<p id="49a9" class="im in ft ip b iq ir hp is it iu ht iv iw ix iy iz ja jb jc jd je jf jg jh ji dc ec" data-selectable-paragraph="">Zyte: The smartest rotating proxy for web scraping</p>
<p id="e4ed" class="im in ft ip b iq ir hp is it iu ht iv iw ix iy iz ja jb jc jd je jf jg jh ji dc ec" data-selectable-paragraph="">Specially designed for web scraping, zyte allows you to crawl quickly and reliably, managing thousands of proxies internally, so you don&rsquo;t have to. You never need to rotate a proxy again.</p>
<p id="9571" class="im in ft ip b iq ir hp is it iu ht iv iw ix iy iz ja jb jc jd je jf jg jh ji dc ec" data-selectable-paragraph="">Since then zyte has undergone numerous redesigns and improvements to keep pace with the changes in web scraping technologies and cope with the ever more complex challenges experienced when scraping the web.</p>