teleren/devto huggingface dataset:
Best Open Source Frameworks for Web Scraping
Published: 2021-04-16 09:19:00

Best Web Scraping Frameworks

Table of contents

PHP
Goutte
Python
Scrapy, Mechanical Soup, PySpider
Language agnostic

Selenium Web Driver
Node Js

Apify SDK
Codecrawler
Puppeteer
Golang

Colly
Goutte
Goutte is a screen scraping and web crawling library for PHP.

Goutte provides a nice API to crawl websites and extract data from the HTML/XML responses.

PHP Version: PHP 7.1 .

Example of submitting a form in Goutte
Scrapy

Scrapy is an extremely powerful crawling and scraping library written in Python.

Here is how easy it is to create a multi-threaded crawler and parse it at a single endpoint.
And to scrape, it allows both XPath and CSS selectors.
Mechanical Soup

Mechanical soup is a super simple library that helps you scrape, store and pass cookies, submit forms, etc. but it doesn't support Javascript rendering.

Here is an example of submitting a form and scraping the results on Duck Duck Go
PySpider

PySpider is useful if you want to crawl and spider at massive scales. It has a web UI to monitor crawling projects, support DB integrations out of the box, uses message queues, and comes ready with support for a distributed architecture. This library is a beast.

You can do complex operations like.

Set priorities.
Set delayed crawls. This one crawls after 30 mins using queues.
This one automatically recrawls a page every 5 hours.
NodeCrawler

This powerful crawling and scraping package for Node Js allows server-side DOM and injection of JQuery and has queueing support with controllable pool sizes, priority settings, and rate limit control.

It's great for working with bottlenecks like rate limits that many websites impose.

Here is an example that does that.
Selenium Web Driver

Selenium was built for automating tasks on web browsers but is very effective in web scraping as well.

Here you are controlling the Firefox browser and automating a search query.
Its language agnostic, so here is the same thing accomplished using Javascript.
Puppeteer

Puppeteer lives up to its name and comes closest to full-scale browser automation. It can do more or less everything that a human can do.

It can take screenshots, render javascript, submit forms, simulate keyboard input,

This example takes a screenshot of the Ycombinator home page in very few lines of code.
Colly

Colly is a super fast and scalable and extremely popular spider/scraper.

it supports web crawling, rate limiting, caching, parallel scraping, cookie, and session handling and distributed scraping

Here is an example of fetching 2 URLs in parallel.
If you’re not living on the edge, you take up too much room
Tags: net, development, leadership, programming
Published: 2021-04-16 10:57:40

---
title: If you’re not living on the edge, you take up too much room
published: true
date: 2021-04-16 09:26:00 UTC
tags: net,development,leadership,programming
canonical_url: https://craignicol.wordpress.com/2021/04/16/if-youre-not-living-on-the-edge-you-take-up-too-much-room/
cover_image: https://craignicol.files.wordpress.com/2020/02/wp-1582925757838.jpg?w=2400
---

.net is a battleship, and it’s taken a long time to change everything to core, and figure out what the Framework/Core future is. In the meantime, you may have found your project crushed in the path as new APIs change or old technology gets deprecated.

Ask Java developers about where Oracle is taking their language and you’ll hear a similar story. The future is different. Maybe better, but in some places definitely worse.

Change is inevitable. That’s why our industry has embraced agile, so we’re ready to change on weekly or monthly cycles, not yearly ones. The longer it takes to make the decision to change, the more baggage you have, and the harder that change will be. That’s why Jez Humble recommends “if it hurts, do it more often”. That change may come from the business, from the competition, from the platform, or from the environment. How did you deal with Heartbleed, or Spectre? How many of your customers are still vulnerable, and are reducing herd immunity for the rest? Are other companies carrying your baggage without knowing it? Are you the reason that IE6 VMs are still a thing?

The bleeding edge is painful. .net Core broke things, Angular 2 broke things, Python 3 broke things, [Edge broke things](https://craignicol.wordpress.com/2017/03/07/microsoft-edge-ungooglability-and-a-new-class-of-bugs/). But not keeping up breaks more.

How do _you_ keep your tech contemporary?
Beautiful Soup vs. Scrapy
Published: 2021-04-16 09:20:22

We always wanted to make a comparison between both of these libraries we have used extensively and even written extensively about.

We love both of them, and sometimes, I fire up one or the other without almost any thought, especially for smaller, quick ones. Scrapy is the one we lean towards when it is a massive web scraping project, and beautiful soup is quicker to get going with for smaller hacks.

Here is how they compare in our opinion
Both of these can be used well combined, in our opinion.

But in production where you want to scale to thousands of links, then you will find that you will get IP blocked quickly by many websites as well. In this scenario, using a rotating proxy service to rotate IPs is almost a must.

Otherwise, you tend to get IP blocked a lot by automatic location, usage, and bot detection algorithms.

Our rotating proxy server Proxies API provides a simple API that can solve all IP Blocking problems instantly.

With millions of high speed rotating proxies located all over the world
With our automatic IP rotation
With our automatic User-Agent-String rotation (which simulates requests from different, valid web browsers and web browser versions)
With our automatic CAPTCHA solving technology
Hundreds of our customers have successfully solved the headache of IP blocks with a simple API.

A simple API can access the whole thing like below in any programming language.

You dont even have to take the pain of loading Puppeteer as we render Javascript behind the scenes, and you can just get the data and parse it any language like Node, Puppeteer or PHP or using any framework like Scrapy or Nutch. In all these cases, you can just call the URL with render support like so...
Best Free & Paid Website Rippers
Published: 2021-04-16 09:19:44

This is one of the questions we get asked very often. Mohan, I need to download an entire website. How do I do it?

These types of tools are called website ripper, website grabber, website downloader, or website crawler, or website spider.

Octoparse

Octoparse is one of the most accessible tools for ripping entire websites for three reasons

It has a wizard which helps you set it up
It has something called Task Templates that give a pre-set template to download oft-used websites like Amazon, Yahoo or Best Buy
It has a desktop and a cloud offering
Download the Windows ripper software here https://www.octoparse.com/Download

SurfOffline

SurfOffline is another desktop tool that works on Windows that rips entire websites onto your disk.

It supports high-speed downloads (up to 100 threads simultaneous) can download millions of web pages and can download images, videos, and documents
Get it here http://www.surfoffline.com/

HTTrack

HTTrack can replicate an entire website on your computer, including the directory structure.

It even modifies the links to point to your offline download so you can browse the website as if you were browsing online.

It can resume broken downloads and even update an existing download.

Moreover, the software is entirely free because it is open source.

It works on Windows/Linux and Unix systems.
Get it here https://www.httrack.com/

WebCopy

WebCopy is similar to the open-source offering above, but it works only on Windows.

WebCopy downloads everything for Offline viewing and can replicate all the CSS and javascript and links and map it to your local copy.


Get it here https://www.cyotek.com/cyotek-webcopy
I made a SandBox coding app!
Tags: html, javascript, css, codepen
Published: 2021-04-16 09:18:02


## I made a Desktop app called [CodeBox](https://github.com/virejdasani/CodeBox).
- It is a coding environment with HTML, CSS and JavaScript editors with a live preview panel to show what the code would look like in the browser
- I made it using the Electron framework
- It is available for all major operating systems
- It is inspired by Codepen and JsFiddle

![CodeBox-Preview](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hxnzeq390mx60u4ty28c.png)
 

## Why use CodeBox?
- CodeBox is better than the editors in CodePen and JSFiddle because CodeBox features the same editor used in VS Code.
- This means that it has autocomplete, syntax highlighting, and a lot of other features that we all know and love from VS Code
- Furthermore, it features hot reload which means, you don't have to press Ctrl/Cmd + S to show the code in the preview panel. It reloads automatically every time any key is pressed!

####- Check it out on GitHub [here](https://github.com/virejdasani/CodeBox)
####- Download it [here](https://github.com/virejdasani/CodeBox/releases/tag/v1.0.0)

> I would really appreciate a star on GitHub and any feedback you may have. Thanks in advance!
Building a Web Crawler? Here Are All The Places That It Will Probably Fail At
Published: 2021-04-16 09:17:51

Here is a list of places that your web crawler will probably fail at. You will need to build in checks for each and also expect them to happen. Send yourself alerts by having portions of the scripts check for unexpected behavior.

If your web crawler is stuck, you need to know
If your web crawler is slowing down, you need to know
if you are having internet issues, you need to know
if the data you are getting is weird, you need to know
Use can also use external tools like these to help keep them up.

Log all the steps your web crawler is taking and the time it took for each. Build in a check where your code sends you an alert when the time has taken is too long and if it 'knows' the data that should be fetched, but it is not fetched this time.

Here is a list of places that you need to pay special attention to in your code to prevent breakages

We the web pages dont load
Internet is down
When the content at the URL has moved
You are shown a CAPTCHA challenge.
The web page changes its HTML, so your scraping doesn't work.
Some fields that you scrape are empty some of the time, and there is no handler for that.
The web pages take a long time to load
The web site has blocked you completely
Building and Testing USSD and SMS Applications complete course
Tags: ussd
Published: 2021-04-16 09:08:43

For a long time, I looked for an online course to learn how to develop and test and deploy USSD applications. I was not successful. I later discovered that building USSD apps is a matter of consuming API but one still has to understand a thing or 2 about GSM networks/mobile telecom networks.

USSD technology is an old technology but it has found a new meaning, and it is being embraced to create cutting edge solutions. 

I embarked on creating a complete course on USSD and its sister SMS :) 

It is an hands-on course, where a learner will have built a simple mobile money application running on USSD. 

The course is made of **152 lessons** and is **12 Hrs 35 minutes** long. It is organized into chapters as follows:

- Course introduction 
- Introduction to USSD and SMS Applications
- Introduction to PHP programming language and HTTP protocol 
- Building and Testing USSD Application Menu
- Adding application functionality or logic to the USSD Menus
- Adding SMS functionality to the USSD application
- Moving from SANDBOX environment to LIVE environment
- Monetizing USSD and SMS Applications 

Access the course using this link: [Building and Testing USSD and SMS Applications](https://skillsday.co/courses/building-and-testing-ussd-and-sms-applications/)
What is sass?
Tags: scss, design, webdev
Published: 2021-04-16 09:08:39

Hello, there
I'm going to explaining to you what scss is
first, we need to know what SCSS is? Sass means Syntactically awesome stylesheet. sass is also a CSS preprocessor used in writing better css  
You may be wondering how to write sass I'm going to show you how, and the best practices
Sass reduces the repetition of CSS and therefore saves time.
`/* Define standard variables and values for website */
$bgcolor: light blue;
$textcolor: dark blue;
$fontsize: 18px;

/* Use the variables */
body {
  background-color: $bgcolor;
  color: $textcolor;
  font-size: $fontsize;
}`
## Why Use Sass?
Stylesheets are getting gigantic, more complex, and harder to maintain, or for open-source contributors to understand. This is where a CSS pre-processor can help you out.

Sass lets you use features that do not exist in CSS, like variables, nested rules, mixins, imports, inheritance, built-in functions, and many more.
##How does SASS work?
You may be wondering this SASS sounds, looks cool but how does it work well the browser doesn't understand scss but it understands CSS this is where an interpreter is also known as a converter arrives converting your scss code into plain ol'css 
we have many interpreters one of them is a live scss compiler in the vscode extension marketplace
## SASS file extension
.scss or .sass but I prefer .scss 
## Sass Comments
Sass supports standard CSS comments like `/* comment */`, and in addition it supports inline comments `// comment`
Insert center image in README.md on Github
Tags: github, readme, markdown, html
Published: 2021-04-16 08:58:49

Github is popular. As far as I see, most of developers know it quite well. They create projects, contribute code, write instructions in README.md and so on.

In README.md, one day you want to insert center image into it. You don't know somehow to do it using Markdown. Fortunatelly, Markdown on Github supports some HTML tags. Two of them are p and img.

The following is the way to insert center image into README.md:

```markdown
<p align="center">
   <img src="[YOUR_RAW_IMAGE_URL]" alt="[YOUR_ALT]"/>
</p>
```

It's cool, right?

Hope you enjoy this post.

Happy coding :)
Let's make some connections!
Published: 2021-04-16 08:51:01

Hi there! 👋
Hope everyone's having a great day. I've recently watched a great interview with @cassidoo, in which she emphasised the importance of making connections. I'm new here, I really don't know anyone, and I'd love to change it!

My name's Kasper, I've just turned 23. I'm a Full Stack Developer @ Ringier Axel Springer Polska (media&tech company). I live and work full time in Poland. I also do university on the weekends, Bachelor in Computer Science. Currently trying to make it all work together and looking for ways to grow as a Software Engineer 🚀

If anyone is interested in sharing their story and making new connections, please DM me or leave a comment, I'd love to reach out to you! 😁
Download file using HttpClient wrapper asynchronously.
Tags: httpclient, dotnet, http, csharp
Published: 2021-04-16 08:46:10

HttpClient is a simple and robust wrapper to send and receive HTTP requests. It's an awesome alternative to the legacy HTTP client .NET api. I like HttpClient the best. It's free, efficient and especially easy to use. You can send HTTP requests and then receive data back only with a couple of code lines.

For instance:
A function to download file with a provided uri and output path.

```csharp
public static class HttpHelper
{
   private static readonly HttpClient _httpClient = new HttpClient();

   public static async void DownloadFileAsync(string uri
        , string outputPath)
   {
      Uri uriResult;

      if (!Uri.TryCreate(uri, UriKind.Absolute, out uriResult))
         throw new InvalidOperationException("URI is invalid.");

      if (!File.Exists(outputPath))
         throw new FileNotFoundException("File not found."
            , nameof(outputPath));

      byte[] fileBytes = await _httpClient.GetByteArrayAsync(uri);
      File.WriteAllBytes(outputPath, fileBytes);
   }
}
```

Hope you enjoy this post.

Happy coding :)
SVG Circle Pattern Generator
Tags: svg, design, javascript, webdev
Published: 2021-04-16 08:32:07

The other day – trying to empty my head and relax a bit – I stared at the [colorful rug](https://hay.dk/da-dk/hay/rugs-7e0ae5e6/pinocchio) next to my chair:

![Hay Rug](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/993bp4vepj8kjd7al2gg.jpg)

And suddenly, instead of emptying my head, I started to wonder about how I could re-create this in `svg` — that is: repeating circles with random colors!

---

When working with *random patterns*, the first thing you need is a random function:

```js
function R(max, min = 0) {
  return Math.floor(Math.random() * (max - min) + min);
};
```
If you call it like `R(360)`, it will return a random number between `0` and `360`. If you add a second parameter, like `R(360, 200)`, it will limit the random number to a number between `200` and `360`.

The *circles* are created within a dual loop – one for the `y`-axis, and one for the `x`-axis (*excerpt*) :

```js
<circle
  cx="${cx}" cy="${cy}" r="${r}" 
  fill="hsl(${R(A.elements.he.valueAsNumber,
 A.elements.hs.valueAsNumber)},
 ${R(A.elements.se.valueAsNumber,
 A.elements.ss.valueAsNumber)}%,
 ${R(A.elements.le.valueAsNumber,
 A.elements.ls.valueAsNumber)}%)"
/>
```

Most of the code deals with the *random colors*.

`hsl()` is perfect for this, since it's three ranges of numbers, that can be randomized:

- hue (0-360)
- saturation (0-100)
- lightness (0-100)

`A` is the main form for editing the variables:

![Form](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vukt49kodfs7c525cbz7.png)
 
– and as mentioned above, you can *limit* the randomness, in this case the `hue` (min and max):

![circles](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5ug8gm0c4h4gjiv90l5e.png)

If you set the `radius` to a larger value than the `size`, you end up with something similar to my rug: 

![Pattern](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dtpr64r08uhucwcasv10.png)

Here's a Codepen with the final result. Open it in [fullscreen](https://codepen.io/stoumann/full/PoWRaaR), click on the rotating `cog-wheel` to open the editor, and have fun!

{% codepen https://codepen.io/stoumann/pen/PoWRaaR %}

If you want, you can save the pattern you create as either `png`, `jpg` or `webp` — just select a type, and click on `Save to Image`.

Thanks for reading!
Which Linux distro / which lineage do you use?
Tags: linux, discuss, fedora, docker
Published: 2021-04-16 08:10:53

I mean, for more than a few months or years or so...

- If you use Ubuntu, or Debian-based; then it is probably a no-brainer?
  - That includes, if you had to compile from source.
- Arch Linux, or Arch-based, like Manjaro, should have quite a large and thorough documentations, because of Arch itself
- What about Fedora, CentOS, OpenSuSE, BSD, Oracle? How do you survive those?
  - Somehow, DaVinci video editor is developed in CentOS.
- Have anyone tried [Darwin](https://en.wikipedia.org/wiki/Darwin_%28operating_system%29)-based?
- What about Alpine, or something else? Do you need a desktop environment at all, if you can survive with CLI, or perhaps, tmux?
- Does it affect you Docker distro of choice?
How can we enter our User Id and Password in a windows Popup using Puppeteer?
Tags: authentication, windowspopup
Published: 2021-04-16 08:02:50

After the page is redirected I was not able to generate a pdf as I was not able to mention my UserId and password in the windows popup dialog box as mentioned in the image.
https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1j4ihsc7fh0f29egednk.png

1.Tried using cookies but didn't get a solution.
2.Tried to use "await page.keyboard.type('xyz@abc.com');" but the text is not getting dumped into the Username field.

How can we enter our details in that dialog box to move to the next page?

Looking for help if we have any other way to enter our details in the windows popup and proceed further?
Next.js and NextAuth.js
Tags: nextjs, nextauth
Published: 2021-04-16 07:47:54

Passwordless Email login using the link in NextJs - [https://lagandlog.com/logs/passwordless-email-login-in-nextjs-using-nextauth](https://lagandlog.com/logs/passwordless-email-login-in-nextjs-using-nextauth)

NextAuth.js is a complete open-source authentication solution for Next.js applications.

It has built-in support for all sign-in options out there (such as Google, Github, Twitter, etc)

Supports both JSON Web Tokens and database sessions.

# Initialization

Create your NextJs project using,

```
yarn create next-app
```
The project will be created and the folder structure will looks like,

```
.
.
.
\pages
--\api
--\index.jsx
.
\package.json
```

Install the next-auth dependency using,

```
yarn add next-auth
```
Create a folder called `auth` inside `/pages/api`

To add NextAuth.js to a project create a file called [...nextauth].js in `pages/api/auth`.

```
import NextAuth from 'next-auth'
import Providers from 'next-auth/providers'

export default NextAuth({
  // Configure one or more authentication providers
  providers: [
    Providers.GitHub({
      clientId: process.env.GITHUB_ID,
      clientSecret: process.env.GITHUB_SECRET
    }),
    // ...add more providers here
  ],

  // A database is optional, but required to persist accounts in a database
  database: process.env.DATABASE_URL,
})
```

List of [other providers](https://next-auth.js.org/configuration/providers)

That's it you have integrated Github authentication in your nextjs application.

# Default APIs

NextAuth provides various APIs inside the library by default,


You can handle the current session of the user effectively using `next-auth/client`

```
// server side
import { getSession } from 'next-auth/client'

// client side hook
import { useSession } from 'next-auth/client'
```

More about [`next-auth/client`](https://next-auth.js.org/getting-started/client)

# REST APIs

NextAuth also provides REST APIs out of the box for,
- GET /api/auth/signin
- POST /api/auth/signin/:provider
- GET /api/auth/signout
- and [more](https://next-auth.js.org/getting-started/rest-api)

Includes TypeScript support.
```
yarn add -D @types/next-auth
```
Works well with and without a database. You can also connect to database ORMs like Prisma.

Prisma ORM Adaptor - [view](https://next-auth.js.org/schemas/adapters#prisma-adapter)

Learn more at [next-auth.js.org](https://next-auth.js.org)

Passwordless Email login using the link in NextJs - [https://lagandlog.com/logs/passwordless-email-login-in-nextjs-using-nextauth](https://lagandlog.com/logs/passwordless-email-login-in-nextjs-using-nextauth)

That's it. Hopefully, it'll be useful. If you wish, follow me on [Twitter](https://twitter.com/sasiKdharan).