teleren/devto huggingface dataset:
5 Rules For Writing A Web Scraper That Doesn’t Break
Published: 2021-06-01 05:47:27

1. Use a crawling and scraping framework like Scrapy:

Don’t try to reinvent the wheel. Frameworks like Scrapy abstract many of the complex functions of web crawling like concurrency, rate limiting, handling cookies, extracting links, using file pipelines, handling broken and different encoding to make life easier.

Scrapy also makes life easier by providing the support of selectors for scraping content.

2. Learn & Use XPath or CSS selectors

Instead of using RegEx or any other custom rudimentary method to get to the data, you want to scrape, using CSS selectors or XPath or a combination of both makes your code more stable. It protects you against arbitrary changes in a website’s code.

3. Scale using Scrapyd and Rotating Proxies

Scrapyd allows you to run multiple spiders at the same time and manage them easily. Combining it with a rotating proxy like Proxies API means you can scale your project to dramatic speeds and break a lot of usage and concurrency restrictions of linear coding without incurring the wrath of usage restrictions or IP blocks.

4. Take measures to counter usage restrictions and IP blocks

You might have finally written the perfect scraper that gets every piece of information, managed pagination, code variances, javascript rendering, etc. but it might all come to naught if you get IP blocked. Rotating proxies like Proxies API is the way to do overcome it. There is no way around it for serious projects of any decent size, frequency, and importance.

5. Put in checks and balances

There are so many failure points in web crawling projects that you have no control over. It’s best to put it a bunch of checks and balances by first identifying them like:

a. Loss of internet connectivity on both ends.

b. Usage restrictions imposed.

c. IP blocks imposed.

d. The target website changes its HTML.

e. The target website is down.

f. Target website issues a CAPTCHA challenge.

g. The target website changes the rules of pagination.

h. The target website uses cookies now.

i. Target website hides content behind javascript.

The author is the founder of Proxies API, a proxy rotation API service.
🦊GitLab Cheatsheet - 1 - Basics of Stages and Jobs
Tags: gitlab, devops, cheatsheet, tutorial
Published: 2021-06-01 05:43:53

This post is the first one of a new set of GitLab Cheatsheet. This speaks about GitLab CI and basics of stages and jobs.

<a href="https://jeanphi-baconnais.gitlab.io/img/cheatsheet/GitLabCheatsheet-1-Basics_of_stages_&_jobs.jpg" alt="1-Bascis of stages and jobs" target="blank">
<img src="https://jeanphi-baconnais.gitlab.io/img/cheatsheet/GitLabCheatsheet-1-Basics_of_stages_&_jobs.jpg" alt="1-Bascis of stages and jobs" />
</a>

I will create others cheatsheets. Tell me if you have comments or suggest about those cheatsheet 😃.
Top 1 way to not mess up your project 
Tags: sideprojects, webdev, programming, productivity
Published: 2021-06-01 05:41:07

If you are a web developer, maker or an entrepreneur who is trying to build a sustainable businesses, 


Here is the Top 1 (yes, it's not a yet another Top 10 article) way to not mess up your project. 



**1. Don't be a perfectionist**



If you really want to be a perfectionist, change the definition 😂



**Perfectionist - someone who takes the perfect next logical step for the project.**

And not someone who keeps tweaking that 1px off margin 🤣


Not convinced yet? Sorry I didn't try to perfect this post 🤷


--- 

Original Tweet - https://twitter.com/onerinas/status/1399596072318640128

---

Update: Wrote about [2 tricks to stay motivated on your side project](https://dev.to/onerinas/2-tricks-to-stay-motivated-on-your-side-project-4a4b)
How To Be A Successful Web Crawling And Data Mining Agency?
Published: 2021-06-01 05:34:58

Having worked on over a hundred Web scrawling jobs of all sorts of scale and complexity and having built technologies that can scale web crawling projects to millions of URLs per day, our knowledge of customers and web crawling agencies, here are some things that could help you boost your success rate.

1. Have a presence on UpWork

a. It is by far the best way to prove your track record. Customers don’t readily believe people when they have tall claims about the kind of projects they have done in the past. It’s best if you have a neutral third party like UpWork keeping a track record of your work, money earned, and also the ratings and reputation.

A complete profile as below:

b. Plus UpWork is excellent for SEO as Google ranks it very high and you will get a steady trickle of clients from here.

2. Re-assure people that Web crawling is one of your primary jobs. Many agencies have it a tag on Upwork along with a million other tags, so it is difficult to get a sense that they do web crawling as well as let’s say website building or mobile app development or that they even care about getting a project like that.

3. Ideally, have a landing page for web crawling where you only talk about that part of your skillset, list your team, list your projects, reviews, links to Upwork profile, etc.

4. Capture visitors using the email address, request for quotes and web-based chat like Drift or Intercom.

5. Talk about the frameworks you use and are familiar with to give a sense of authority and confidence to your end customers. Even if the end customers don’t know anything about Scrapy or Puppeteer, talking about them gives them a sense of the depth of your knowledge.

6. Openly talk about how you overcome risks like rate limitings, code changes on the target websites, IP blocks. Talk about how you monitor and debug your spiders etc.

7. Blog about some of the key concepts from a layman’s perspective makes YouTube videos to build confidence and also SEO juice to your website.

8. Answer questions on forums like quora by being generally helpful and build goodwill which will inevitably turn to visitors and clients in the long run.

9. Partner with software companies like us to get deal flow. We are Proxies API, a Rotating Proxies API Service that serves hundreds of clients, and we get regular requests for recommendations of agencies that can handle entire web scraping projects all the time. Reach out to us.

The author is the founder of Proxies API, a proxy rotation API service.
5/31 TIL: CGO & GOOS in Go, Github Issue Template, Mathematical Induction
Tags: todayilearned
Published: 2021-06-01 03:50:55

CGO & GOOS in Go

- Cgo enables the creation of Go packages that call C code
- Set CGO_ENABLED=0 or 1 along with Go build to disable or enable the cgo tool
- You configure the Go Operating System using GOOS
- Example of a go build within a Dockerfile:

```

RUN CGO_ENABLED=0 GOOS=linux go build -o fileName .

```

Github Issue Template

- [https://docs.github.com/en/communities/using-templates-to-encourage-useful-issues-and-pull-requests/configuring-issue-templates-for-your-repository](https://docs.github.com/en/communities/using-templates-to-encourage-useful-issues-and-pull-requests/configuring-issue-templates-for-your-repository)

Go Vendor

- [https://blog.gopheracademy.com/advent-2015/vendor-folder/](https://blog.gopheracademy.com/advent-2015/vendor-folder/)

—Discrete Math—

Quotient-Remainder Theorem:

For all n ∈ ℤ, d ∈ ℤ+, there exists q,r ∈ ℤ so that n = dq + r with 0 ≤ r < d

Modular Arithmetic

- n mod d; system of arithmetic where we are concerned with the remainder
- Ex) Clock (n/12 = w + r), Day of the Week (n/7 = w + r)

Sequence

- informal: an infinite ordered list of objects
- formal: a function f: ℤ+ → C

Mathematical Induction

- Prove: P(n), ∀n ≥ a
- Step 1: Prove that P(a) is true—basis step
- Step 2: Assume P(k) is true, prove P(k+1) is true (for k ≥ a )—induction step
Write for Us

Published: 2021-06-01 04:03:00

Smart Tech Data blog for all invited contributors and contributors. We offer “Write for Us” as an opportunity to write your well-researched articles on the latest technology news and trends, laws and business ideas, recently released Gadget Reviews and “Thoughts on Gadgets, Write for Us”, Marketing Tips, Internet of Things (IoT) … ), artificial intelligence (AI) and the promotion of many small companies and industries.

Guest contributors and bloggers are always welcome to share their thoughts and ideas in the Smart Tech Data by contacting us at Contact@smarttechdata.com.

Topics Write To Us: Technology, Business, Marketing, Gadgets
Technologies Write to us
Artificial Intelligence

Amazon Web Services (AWS)

Blockchain

Augmented reality and virtual reality

Cloud computing

Corner and React

DevOps

Data science

Android Application Development

Internet of Things (IoT)

Smart Apps

Large amount of data

RPA (robotic process automation)

Cybersecurity

Business Write to us
Business management

Business strategy

Digital agencies

Strategic management

Small and Medium Enterprises (SMEs)

Social entrepreneurship

E-commerce

Entrepreneurs, Young Entrepreneurs

Small Business Marketing

Startups, Startup Ideas, Startup Consulting & Strategies, Startup Business Models

Ideas for business

Tech startups

Marketing | Writing digital marketing articles for us
Affiliate Marketing, B2B Marketing, Content Marketing, Email Marketing, Digital Marketing

Advertising, Online Advertising, Mobile Advertising, Display Advertising

Web Marketing, Online Video Marketing, Web Analytics

Content strategy, product branding, press releases

Link building, local search, pay per click

Internal SEO, SEO tools,

Search Engine Marketing (SEM), Search Engine Optimization (SEO)

Telecom Writes For Us
Telecommunication networks

5G, 4G, 3G

To turn

Networks

Cloud communication

Voip

Broadband access

Wi-Fi

Gadgets Write To Us
Electronic accessories

Technical gadgets

Spy gadgets

Gadgets for games

Reviews and ratings

Article Requirements To Write For Us For Smart Tech Data
The minimum article length must be more than 700 words and cannot be maximum. Prefer over 1000 words for better rankings.
Unique HD quality image associated with the subject. The size should be 720 X 480 pixels, which should be published along with your article. Images must be in JPEG format.
There should be no duplicate content, and the content should be original and should not have been previously published.
There is no need to add internal links to the article, we will do it.
You can add an external link related to the topic.
We do not accept links or publications of any type related to casinos, gambling or any other illegal articles. @ <a href="https://www.smarttechdata.com/write-for-us/">smarttechdata</a>
Moving Repository from Bitbucket to GitHub
Tags: github, bitbucket, repository
Published: 2021-06-01 04:00:16

I’m working as a Volunteer in a new product/project, ProInstinct, and we needed to move our repositories from Bitbucket (private) to Github (private). Why did we need to move?

Very simple, our team of volunteer developers has grown, and Bitbucket only allows five free members on the team. With the recent move by Github to allow free private repositories and unlimited members, the decision to change was straightforward.

I was a bit uncertain about the sequence of git commands that I should perform without having any problem. After brief research and a few tests with a dummy repository (always test with dummy code first), I realize that it is relatively safe and comfortable.

The first step to ensure that you do not lose all or part of your hard work is to merge all branches. You should have only one branch, master, to do this migration. Clone the repository to your hard drive.

You probably already have your GitHub account, now create your new repository on Github, and save the URL, and follow these steps:

`cd $repository`
`git remote rename origin bitbucket`
`git remote add origin https://github.com/{new repository}.git`
`git push origin master`

This process can take a while; it will depend on the amount of file and size and your internet connection. Once the process is finished, check that your code is all in our repository. Once you are satisfied with sending to GitHub, you can delete the old remote by issuing:

`git remote rm bitbucket`

I spent more time researching and studying how to transfer than doing. Do not forget the most sensitive part of this process are the branches; if you have several branches, the transfer will become more laborious.

But for a single branch, it is straightforward in the end.
add phpinsights in your laravel project
Published: 2021-06-01 01:51:58

phpinsights is a helpful package to add code quality analysis, it measure the code in four areas:

- Code (code reliable, loosely coupled, simple, and clean).

- Complexity (check the statements and methods complexity).

- Architecture (check the patterns and directories that you add, traits, interfaces, etc).

- Style (check how you follow PSR-4 or PSR-12).

So it makes sense right?, lets install it.

## How to install it

```
composer require nunomaduro/phpinsights --dev
```

## Publish config file

```
php artisan vendor:publish --provider="NunoMaduro\PhpInsights\Application\Adapters\Laravel\InsightsServiceProvider"
```

## Execute PHPinsights

```php
php artisan insights
```

### Customize it

On Laravel there are some code standards that could mismatch on how PHPInsights works, so you can edit the config file to adjust to your project/needs/packages etc.

#### Here an example of how I use it with a Laravel project with Jetstream.

You can exclude files by typing the full path name, as an array item.

On `config/insights.php`:

```php
    'exclude' => [
        'app/Actions/Jetstream',
        'HandleInertiaRequests.php',
        '_ide_macros.php'
    ],
```

You can add custom rules:

```php
    'add' => [
        Classes::class => [
            ForbiddenFinalClasses::class,
        ],
    ],
```

You can even remove rules that does not make sense, you can customize it as you need:

```php
    'remove' => [
        AlphabeticallySortedUsesSniff::class,
        DeclareStrictTypesSniff::class,
        DisallowMixedTypeHintSniff::class,
        ForbiddenDefineFunctions::class,
        ForbiddenNormalClasses::class,
        ForbiddenTraits::class,
        ParameterTypeHintSniff::class,
        PropertyTypeHintSniff::class,
        ReturnTypeHintSniff::class,
        UselessFunctionDocCommentSniff::class,
        UnusedParameterSniff::class,
        LineLengthSniff::class,
        DocCommentSpacingSniff::class,
        ClassInstantiationSniff::class,
        NewWithBracesFixer::class,
        NullableTypeForNullDefaultValueSniff::class,
        DisallowArrayTypeHintSyntaxSniff::class,
        NoEmptyCommentFixer::class,
        DisallowShortTernaryOperatorSniff::class,
        ForbiddenPublicPropertySniff::class,
        DisallowEmptySniff::class

    ],
```

Another cool feature that you can customize is the "level" of qualification to print it as "green", "yellow" for example, you can customize it on `requirements` key, but if you comment, all the content it will work as it is designed by default:

```php
    'requirements' => [
//        'min-quality' => 0,
//        'min-complexity' => 0,
//        'min-architecture' => 0,
//        'min-style' => 0,
//        'disable-security-check' => false,
    ],
```

This are some useful customizations, if you want to run automatically `PHPInsights` on every commit or push you can install husky and configure it to run PHPInsights on the husky hooks, is just a cool idea to make your work easier.

That is all for this post, thanks for reading!
[Resolved] MongoDB Atlas line-break issue
Tags: programming, javascript, mongodb
Published: 2021-06-01 03:33:57

# The context
While creating [this](https://mynoteit.herokuapp.com) PWA, I wanted to store the *markdown* data into the MongoDB's [Atlas](https://www.mongodb.com/cloud/atlas).

**Example Markdown**

```markdown
# Some interesting title
Description about the topic...

- list #1
- list #2

> Maybe a quote ?
```

The above markdown would be stored as a **single string** in the database.

```js
{
  // other fields
  markdown: "# Some interesting title\nDescription about the topic...\n- list #1\n- list #2\n> Maybe a quote ?"
  // further fields
}
```

# The problem

While reading the data from the [Atlas](https://www.mongodb.com/cloud/atlas), the line-break escape character, i.e. `\n` would come as already escaped, i.e. `\\n` **notice the double '\\'**.

Therefore, while parsing it as *HTML*, the **line-break** wouldn't be read as a line-break character but a literal *\n* character.

**Rendered HTML**

![Unexpected Render of Markdown](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/r9145c0srdckpb11chax.png)

*The Markdown parser([marked.js](https://marked.js.org/)) expects a line-break between each block(headings, lists, paragraphs, quotes, etc) to render them as expected. Otherwise, it will render them as a single line of string.*

*In the example above, it renders everything as a heading level 1.*

## Why ?

When the Markdown parser sees `# ` token, it assumes that the text after it(until a line-break) is to be rendered as a H₁. Thus, everything including lists, paragraphs, quotes are rendered as `<h1>` because of no line-break.

# The Solution

I made a mistake by thinking that the problem was with the *Markdown parser,* while instead the problem was with the **data** coming from MongoDB.

The doubly escaped character `\\n` was the culprit.

As a workaround, I tried to *replace* all `\\n`s with `\n`s  from the markdown string before passing it to the [Markdown Parser](https://marked.js.org/).

```js
import parser from "marked";

// 'markdownString' would be the markdown field read from mongodb
const replacedWithSingleEscape = markdownString.replace(/\\n/g, "\n");

parser(replacedWithSingleEscape);
```

**Solved! This is how the rendered output would look like after the fix**

![Expected Render of Markdown](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8mefwctfjk9u3hcq3kfj.png)
Angular : How to automate angular deployment for AWS S3 Angular Website??
Tags: angular, typescript, programming, webdev
Published: 2021-06-01 03:31:38

If you have hosted your angular app to AWS S3 then you can automate your ng build -prod and drag and drop to S3 bucket by simply following below steps -  
  
**Step 1 - Add @jefiozie/ngx-aws-deploy to your project and for this run below command (your angular CLI should be up-to-date)**


```
ng add @jefiozie/ngx-aws-deploy
```

**Step 2 - Provide necessary details to the terminal while step 1 is in progress.**  
You will be prompted for a couple of questions:  
Your AWS Region  
The bucket you would like the files to be uploaded.  
The Secret Access Key  
The Access key Id  
The folder where the files should be uploaded (optional)  
  
**Step 3 - Verify the details you have provided is correct, open angular.json and search for deploy and should be able to see below code** -

```javascript
angular.json is update with a new builder:

"deploy": {
    "builder": "@jefiozie/ngx-aws-deploy:deploy",
    "options": {
    "region": "YOUR REGION",
    "bucket": "YOUR BUCKET",
    "secretAccessKey": "YOUR SECRET ACCESSKEY",
    "accessKeyId": "YOUR ACCESS KEY ID"
    }
```

You can get your access key id and secret access key in your AWS console. Region you can see in your S3 url and bucket name you already know but if you can see it in your url.  
  
**Step 4 - Run ng deploy command and it will build and deploy your app to S3**.  
  
All Done  
  
**Extra Bits-**    
If after following all above steps you run into trouble and get below error -

```javascript
403 Forbidden
Code: AccessDenied
Message: Access Denied
```

Then the solution is to specify bucket policy, for this go to S3 and then your bucket and then permission, there you will find bucket policy, just paste below code and click save.  

```javascript
{ "Version": "2012-10-17", "Statement": [{ "Sid": "PublicReadGetObject", "Effect": "Allow", "Principal": "*", "Action": [ "s3:GetObject"], "Resource": ["arn:aws:s3:::your-bucket-name-here/*"] } ]}
```

In above code replace the word your-bucket-name-here with your bucket name and click on save.  
  
All done, refresh the page and your app will up and running.


<a href="https://www.buymeacoffee.com/rkycom" target="_blank"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" style="height: 60px !important;width: 217px !important;" ></a>

With all that being said, I highly recommend you keep learning!

Thank you for reading this article. Please feel free to connect with me on [LinkedIn](https://www.linkedin.com/in/www-rajeshkumaryadav-com/) and [Twitter](https://twitter.com/rky_com).
Notice
Published: 2021-06-01 03:19:31

To           All employees
From         Desiree Yeung
Date         Oct. 27, 2030
Subject      Annual Physical Exam


Please note that the annual physical exam will be held inside the company office instead of a satellite clinic. This is to ensure the safety and security of everyone in Quicklight Pharmaceuticals.

Services will start at 7 AM. Please bring both HMO and company ID and fill up the digital registration form that will be sent to your email accounts.


189 49th Avenue
Grise Fiord, NU X0A 0J0
www.reallygreatsite.com
(098) 765-4321
official Notice
Published: 2021-06-01 03:15:04

>To          Marketing Staff
>From        Desiree Yeung
>Date        Oct. 27, 2030
>Subject     Social Media Branding


`The company has gone through significant growth in the past few months. As such, a few of our campaigns for the upcoming year needs to be reviewed and re-aligned.`

`One area that we want to focus on is the company's social media branding. While we have a strong following and a distinct brand identity, we want to be able to grow that and stay relevant for far longer.`

`The meeting with the  marketing staff aims to hear ideas and suggestions on how we can move to this direction.`

`The meeting will be on November 5, 10 AM, Seelie Hall.`
`


DLY Urban Club
123 Anywhere St., Any City ST 12345
hello@reallygreatsite.com
Memo
Published: 2021-06-01 03:09:15

To          All employees
From        Ndemi Otieno
Date        23 November 2030
Subject     Bank account change

Team,

Please be informed that the company will be changing its bank from Golden Ore Bank to Zenith Central Bank.

You will receive an email from finance within the week to facilitate this change. Please fill out the forms before 25 November 2030 so we can process the application for you. Any submission after that will have to be facilitated by the employee personally.

Please don't hesitate to reach out if you have any questions.

Sincerely,


Ndemi Otieno
Head of Human Resources

                                                2507 Snowbird Lane
                                                Bellevue, NE 68005
                                           www.reallygreatsite.com
Official Memo
Published: 2021-06-01 03:03:46

TO         UX/UI Team
From       Cia Rodriguez
Date       November 28, 2020
Subject    Updated Work from Home Guidelines



Due to the increasing number of infected cases in the city, we are extending our Work from Home setup until further notice.

Please reach out to our office facilities if you need any of equipment from the office. We can arrange a delivery for you in the soonest possible time.

We are on flexi-time, but please ensure you are online and on time for meetings. Coordinate with your teams to create timelines especially if you have projects that are urgent.

We are here for you. Stay safe.