teleren/devto huggingface dataset:
VM-like containers with Docker + Sysbox
Tags: docker, devops, kubernetes, security
Published: 2020-11-02 17:38:27

Anyone looking to deploy containers that can act like VMs and run software such as systemd, Docker, and even Kubernetes, take a look at the open-source [Sysbox](https://github.com/nestybox/sysbox) runtime.

It enables deployment of "VM-like" containers easily and securely (without using privileged containers). It works under Docker (it's a new type of runc), so deploying such a container is as simple as:

"docker run --runtime=sysbox-runc -it some_image".

You can use these for dev, testing, CI/CD, and to replace heavier VMs. 

Would love to hear if you find this useful or not.
AWS Lambda and API Gateway
Tags: webscraping
Published: 2020-12-02 15:41:10

---
title: AWS Lambda and API Gateway
published: true
date: 2020-11-02 17:26:20 UTC
tags: WebScraping
canonical_url: https://javascriptwebscrapingguy.com/aws-lambda-and-api-gateway/?utm_source=rss&utm_medium=rss&utm_campaign=aws-lambda-and-api-gateway
---

We have a video for you!

<iframe title="Custom domains and AWS API Gateway struggle. SOLVED!" width="640" height="360" src="https://www.youtube.com/embed/9PbzxFMRHyE?feature=oembed" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

This discusses using AWS Lambda with API Gateway. It addresses using custom domains and getting them tied together.

I struggled with this and this video contains the solution that I used.

The post [AWS Lambda and API Gateway](https://javascriptwebscrapingguy.com/aws-lambda-and-api-gateway/) appeared first on [Javascript Web Scraping Guy](https://javascriptwebscrapingguy.com).
Create Web Crawlers That Don't Die on You

Published: 2020-11-02 17:24:25

Here are some rules of thumb to follow when building web crawlers that can scale.

De-couple the web crawling and the web scraping process. This is because you can then measure and speed up the performance of each of these processes separately.
Do not use Regex for scraping. Use XPath or CSS selectors instead. Regex will be the first to break when the target web page's HTML changes even a little bit.
Assume every external dependant process or method will fail and write handlers and loggers for each. For example, assume the URL fetch will fail, timeout, redirect, return empty, or show a CAPTCHA. Anticipate and log each of these exceptions.
Make it easy to debug your app by tracing all the steps your crawler goes through. Make the logger as rich as possible. Send yourself alerts so you immediately know if there is something wrong.
Learn how to get your crawlers to pretend to be human.
Build a crawler around a framework. Custom code will have a bunch more points of failure.
Do you make these 9 mistakes while web crawling?
Published: 2020-11-02 17:23:16

Here is a bunch of things that can get you in trouble while web crawling:

Not respecting Robots.txt.
Not using Asynchronous connections to speed up crawling.
Not use CSS selectors or XPath to reliably scrape data.
Not use a user-agent string.
Not rotate user-agent strings.
Not add a random delay between requests to the same domain.
Not use a framework.
Not monitor the progress of your crawlers.
Not use a rotating proxy service like Proxies API.
Being smart about web crawling is realizing that it's not about the code. In our experience at Teracrawler developing cloud-based web crawlers at scale, most of the web crawling and web scraping is about controlling these variables. Having a systematic approach to web crawling and getting to a place where you can get frequent and reliable data and scale day in and day out can change the fortunes of your company.
Newman integration in the QA process
Tags: testing, tips, api, engineeringmonday
Published: 2020-11-02 17:21:58

__Newman__ is a command line Collection Runner for the Postman and we use it in order to continually test API calls on the backend which are in complex forms involving various filters and tags which are sent in URL parameters.

While using Postman Interceptor in browser, all the requests that are called on as a result of using and clicking on the app itself, and which are turned into collections, are sent on the Postman app.

Collections, with the usage of Newman through command line, can run as a simulation of calling up all the requests from a given collection.

###__Setup__
__Postman Interceptor__
* Visit the Chrome Web Store and download the interceptor as an extension
* Click on the satellite icon which showed up in browser. Afterwards, click on the Install Interceptor Bridge
on an opened popup window.

__Collection preparation__
* Open the Postman app or download it from the web (for an app to work with the interceptor, Postman must be in a desktop version)
* Open the app you wish to test in the browser in an additional tab.
* Open the Postman Interceptor on that exact tab and turn the Capture requests option on.
* All the requests, which are sent from the web app in history tab, will be received on a desktop version of
the Postman app.
* Mark all the requests which are relevant for automatic continuous testing then save them in a
collection.
* Go to Collections tab and click &#39;edit&#39; above the created collection
 * Open the tab Tests on an open popup window and write the following: 
```python
pm.test("Status code is 200", () => {
  pm.expect(pm.response.code).to.eql(200);
});
```
* Go to Collections tab and click &#39;export collections&#39; to export the created collections.
Collections are exported in Collection v2.1 (recommended form) and locally saved.

__Project Preparation__
* Open the project in a code editor
* Installation:
```python
npm install -g newman
```
* Create a folder &#39;newman&#39;
* Move the collection for that project from the local machine to a newman directory.
* Run:
```python
newman run newman/mycollection.json
```
* Create a script in package.json:
```javascript
"scripts": {
    "newman:run": "newman run newman/mycollection.json"
  },
```
__CI/CD__
For this part, u need to choose your preferred tool for CI/CD and create necessary files in the project example JenkinsFile for Jenkins where it's described where and when will your script run.
Hacktoberfest! Last PR! Dutch National Flag~
Tags: opensource
Published: 2020-11-02 17:20:23


*Every party must come to an end.* This is the last PR that I made for Hacktoberfest. For the first time that I realize how fun it was. To be honest, each pull request literally pulls you up.

My last PR is not something big. I was just doing something ordinary my self. I worked on [Dutch National Flag problem](https://en.wikipedia.org/wiki/Dutch_national_flag_problem). The problem is there is an array consisting only 0, 1 and 2. How to sort the array in the linear time complexity.
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/83j2ccmr50n9v1l7tdfs.png)
Well, the problem did raise the curiosity in my head which makes me want to try it. Here is [my solution](https://github.com/aditya109/git-osp-for-beginners/pull/348):

```
 public static void SortingArray(int a[]){
        // Count the number of zero, one
        int zeros, ones, k;
        zeros = ones = k = 0;

        for (int i = 0; i < a.length ; i++){
            if (a[i] == 0 )
            {
                zeros++;
            }
            if (a[i] == 1 )
            {
                ones++;
            }
        }

        // fill zero in the first positions of the array
        while(zeros-- != 0){
            a[k++] = 0;
        }

        //fill one in the next empty position
        while (ones-- != 0){
            a[k++] = 1;
        }

        // fill twos in the left positions
        while (k < a.length){
            a[k++] = 2;
        }
        System.out.println(Arrays.toString(a));

    }
```

I would like to explain a bit about my way to approach the problem. To sort the array that only contains 0, 1 and 2, I first count the number of 0 and 1 in that array. Then I over-write the existing array of which the n first positions are filled with n zeros. The next m positions are filled with m ones. The left positions are filled with twos. Therefore, the array is sorted in linear time complexity.

My last PR is not something significant but I would like to share it to you guys. Every `git push` I have made is a push for me to get out of my comfort zone and started to be more proactive myself.
I did not make a good end for Release 0.2
Tags: opensource
Published: 2020-11-02 17:15:14

I was not good at open tasks. This October had been a little frustrating for me. The issues that I can find are either too easy or too hard for me to accomplish. For the last one, I found an issue that needs help with unit tests, the link is [here](https://github.com/konveyor/move2kube-api/issues/21). I was planning to work on this two weeks ago, but it was a little challenging for me, because it should be written in Golang, and I have little knowledge about unit test yet. 

To be able to accomplish this issue, I read the whole tutorial of Golang unit testing and checked several simple examples as well. I thought I should be ready, and I went back to work on the issue, it was depressing that I cannot understand the purposes of most of the functions, and I have no idea where I should put my hands to begin. I contacted the owner asking for any useful advice, and he was kind to ask his colleagues for high-level guidance. I relied all my hope on this possible guidance, the sad thing is that guidance has never shown up since that colleague never responded. 

I fell into slight depression and anxiety depression. Why did not I try to find new issues? I tried, just cannot find something satisfying, the hard ones are not for me to solve, and the easy ones seem not worth the time and energy, and more like the first PR thing rather than the last one. 

I tried hard and I found this one [https://github.com/alexellis/k3sup/issues/279](https://github.com/alexellis/k3sup/issues/279). I started working on it and when I've almost done with it, the owner responded saying that this was assigned to someone else and hoping me to working on a different one... although they didn't start a conversation under the issue... So... I had to start working on another issue [https://github.com/inlets/inletsctl/issues/81](https://github.com/inlets/inletsctl/issues/81), and I made my PR [https://github.com/inlets/inletsctl/pull/82](https://github.com/inlets/inletsctl/pull/82), which is still waiting to be checked and merged.
Scraping Hotel Listings from Booking.com with Python and BeautifulSoup

Published: 2020-11-02 17:14:07

One of the biggest applications of Web Scraping is in scraping hotel listings from various sites. This could be to monitor prices, create an aggregator, or provide better UX on top of existing hotel booking websites.

Here is a simple script that does that. We will use BeautifulSoup to help us extract information and we will retrieve hotel information on Booking.com.

To start with, this is the boilerplate code we need to get the Booking.com search results page and set up BeautifulSoup to help us use CSS selectors to query the page for meaningful data.

We are also passing the user agent headers to simulate a browser call so we don't get blocked.

Now let's analyze the Booking.com search results for a destination we want. This is how it looks.

And when we inspect the page we find that each of the items HTML is encapsulated in a tag with the class sr_property_block.


We could just use this to break the HTML document into these cards which contain individual item information like this:

And when you run it...

You can tell that the code is isolating the cards HTML.

On further inspection, you can see that the name of the hotel always has a sr-hotel__name class... Let's also get the number of reviews, pricing, and ratings while we are at it.

We have also tried to get the Hotel image and link, all crucial pieces of information.

The whole code looks like this..

And when run.

Produces all the info we need.



In more advanced implementations you will need to even rotate the User-Agent string so Booking.com cant tell its the same browser!

If we get a little bit more advanced, you will realize that Booking.com can simply block your IP ignoring all your other tricks. This is a bummer and this is where most web crawling projects fail.

Overcoming IP Blocks

Investing in a private rotating proxy service like Proxies API can most of the time make the difference between a successful and headache-free web scraping project which gets the job done consistently and one that never really works.

Plus with the 1000 free API calls running an offer, you have almost nothing to lose by using our rotating proxy and comparing notes. It only takes one line of integration to its hardly disruptive.

Our rotating proxy server Proxies API provides a simple API that can solve all IP Blocking problems instantly.

With millions of high speed rotating proxies located all over the world,
With our automatic IP rotation
With our automatic User-Agent-String rotation (which simulates requests from different, valid web browsers and web browser versions)
With our automatic CAPTCHA solving technology,
Hundreds of our customers have successfully solved the headache of IP blocks with a simple API.

The whole thing can be accessed by a simple API like below in any programming language.


We have a running offer of 1000 API calls completely free. Register and get your free API Key here.
Scraping Listings from AirBnB with Python and BeautifulSoup

Published: 2020-11-02 17:08:56

One of the biggest applications of Web Scraping is in scraping hotel listings from various sites. This could be to monitor prices, create an aggregator, or provide better UX on top of existing hotel booking websites.

Here is a simple script that does that. We will use BeautifulSoup to help us extract information and we will retrieve hotel information on Airbnb.com.

To start with, this is the boilerplate code we need to get the Airbnb.com search results page and set up BeautifulSoup to help us use CSS selectors to query the page for meaningful data.

We are also p[assing the user agent headers to simulate a browser call so we don't get blocked.

Now let's analyze the Airbnb search results for a destination we want. This is how it looks.

And when we inspect the page we find that each of the items HTML is encapsulated in a tag with the attribute itemprop with the value itemListElement.


We could just use this to break the HTML document into these cards which contain individual item information like this.

And when you run it...

You can tell that the code is isolating the cards HTML.

On further inspection, you can see that the name of the bed and breakfast always have the property aria-label. So let's try and retrieve that.


That will get us the names.

Bingo!

Now let's get the other data pieces..

And when run.

Produces all the info we need including ratings, reviews, link, price, and discounted price.

In more advanced implementations you will need to even rotate the User-Agent string so Airbnb cant tell its the same browser!

If we get a little bit more advanced, you will realize that Airbnb can simply block your IP ignoring all your other tricks. This is a bummer and this is where most web crawling projects fail.

Overcoming IP Blocks

Investing in a private rotating proxy service like Proxies API can most of the time make the difference between a successful and headache-free web scraping project which gets the job done consistently and one that never really works.

Plus with the 1000 free API calls running an offer, you have almost nothing to lose by using our rotating proxy and comparing notes. It only takes one line of integration to its hardly disruptive.

Our rotating proxy server Proxies API provides a simple API that can solve all IP Blocking problems instantly.

With millions of high speed rotating proxies located all over the world,
With our automatic IP rotation
With our automatic User-Agent-String rotation (which simulates requests from different, valid web browsers and web browser versions)
With our automatic CAPTCHA solving technology,
Hundreds of our customers have successfully solved the headache of IP blocks with a simple API.

The whole thing can be accessed by a simple API like below in any programming language.

We have a running offer of 1000 API calls completely free. Register and get your free API Key here.
Election Week distractions?
Tags: discuss
Published: 2020-11-02 17:08:50

---
title: Election Week distractions? 
published: true
description: Election week is going to be interesting. Let's go to the Winchester, have a nice cold pint, and wait for this all to blow over. What are you going to distract yourself with?
tags: discuss
//cover_image: https://direct_url_to_image.jpg
---

There's many times in your life where the only thing you can do is wait. Maybe it's a test exam. Maybe it's your partner giving birth. Or maybe it's waiting for your the party to start/end at awkward Family Thanksgivings.

For Americans, this week is going to be a long one.

Because after voting -- all we can do is wait.

So the question I'd love to throw out --- 

This week: what are you distracting yourself with?
Release 0.2.3
Tags: opensource
Published: 2020-11-02 17:02:25

For the third issue of release 0.2, I found a simple one that requires an HTML page. The issue is here: [https://github.com/consolelancer/portfolio-shala/issues/3](https://github.com/consolelancer/portfolio-shala/issues/3)
I checked and was sure I should be able to complete this, although it has been a long time since the last time that I wrote HTML. 

The PR is here: [https://github.com/consolelancer/portfolio-shala/pull/11](https://github.com/consolelancer/portfolio-shala/pull/11).
Be Selfish, For Once!
Tags: webdev, productivity, advice
Published: 2020-11-02 17:00:42

> This is an ongoing series on how to "finish your projects." Each post dives into a single lesson I've learned that has helped me do better at staying on track and reaching the colloquial finish-line of a project. These lessons are not fool-proof as my [first post in the series](https://dev.to/indoor_keith/we-all-suck-at-finishing-projects-4n6) will tell you, but they can help with morale and reduce the frequency of dropped projects.

Stop trying to make apps that will ~*change the world*~! While the sentiment is heartwarming and commendable, if you're finding it hard to finish your projects, chances are you're putting too much pressure and responsibility on yourself. Not everyone can be Batman.<sup>[1](#batman)</sup> Just like with those lil yellow party cups on airplanes, you need to put yours on before you can worry about everyone else. You need to get better at completing projects before you can take on the world, my fellow disrupters. 

One great way to make sure you follow through with your projects is to pick up projects that *you* would actually use. Think about ways you can you make *your* life easier, more convenient, or at least more entertaining. [@NikkitaFTW](https://twitter.com/nikkitaftw), developer at [@codesandbox](https://twitter.com/codesandbox), made a website that is just a compilation of memes that people made out of photos of her. She did this because she was tired of trying to find them on her own and wanted a one-stop-shop to grab one whenever she wanted to use for whatever. You can find the work of art [here](https://meme.iamsaravieira.com/). This wasn't an industry shaking, bleeding edge, ground breaking project. It was literally just something she thought would be hilarious, so she did it! Working on something that's *for you* will provide you way more motivation to finish than making another note taker or quiz app. At the very least, you'll be doing everyone a service by not adding another quiz app into the webiverse.<sup>[2](#quizapp)</sup>

So get selfish and make that browser extension that deletes all the "blogginess" out of recipes you find online and gets straight to the ingredients. Or make that text generator that takes whatever you write and converts it to how [Tommy Wiseau](https://www.youtube.com/watch?v=2zWOPCi-PWA) would say it. Chances are, a lot of people may love what you thought only you would find amusing.

<a name="batman"><sup>1</sup></a> And why would you even want to be Batman? The guy is sad, *all* the time. 😬

<a name="quizapp"><sup>2</sup></a> That's a joke. Quiz apps are great and if you're just getting started with web development or learning a new framework, making a quiz app is a great way to get familiar with a lot of the essentials. Hell, at the time of writing this, I'm in the middle of making a quiz app in Vue.js 🤣
Week Plan 11/1/2020
Published: 2020-11-02 16:37:23

Week Plan:
* Easy Algorithm Problems
  * https://leetcode.com/problems/is-subsequence/
    [Status Not complete: Moving to Next Week]
  * https://leetcode.com/problems/fibonacci-number/
    [Status Not complete: Moving to Next Week]

* Azure Fundamentals (AZ-900)
  * https://docs.microsoft.com/en-us/learn/paths/explore-microsoft-azure-cloud-concepts/
[Status: Completed remaining modules]
https://dev.to/thelsk7/az-900-cloud-concepts-cloud-services-5gp4

**Summary**:
 * Completed Pending Azure Cloud concepts modules.
 * Algorithm problems is still not completed. Got distracted due to Elections and other personal work.

**Kaigen**:
 *  Avoid distractions and work on the weekly goal. Try to spend at least 30 minutes to 1 hour each day on the weekly goal.
HACKTOBER 2020
Tags: hacktoberfest
Published: 2020-11-02 16:21:01


This is my first hacktober last year also I missed it. But I didn't think that I'll do it. 
As a student, my friends pushed me to be a part of this so I agreed with them. Actually, I didn't have an idea about what to do. So I talked with friends so they help me to find PRs. Then I did what I can do. Some of them are simple. However I'm happy and ill do more contribution in future
2 columns convert into the same one column using python r pandas ..... 
Tags: python, showdev, database
Published: 2020-11-02 16:06:37

HII 
I have 2 columns or 2 lists so I want to make 2 columns into 1 column with updated values.
x1 = [0,15,10,0,10]
x2 = [0,15,0,11,10]
i want to update x2 to xl using python or pandas
output be like
x1=[0,15,10,11,10]
can anyone help me
Daily Challenge #302 - setAlarm
Tags: challenge
Published: 2020-11-02 15:54:52

---
title: Daily Challenge #302 - setAlarm
published: true
series: Daily Challenge
description: Write a function named setAlarm which returns true if you are employed and not on vacation. 
tags: challenge
//cover_image: https://direct_url_to_image.jpg
---
Write a function named setAlarm which receives two parameters. The first parameter, `employed`, is true whenever you are employed and the second parameter, `vacation` is true whenever you are on vacation.

The function should return true if you are employed and not on vacation (because these are the circumstances under which you need to set an alarm). It should return false otherwise. Examples:
```
setAlarm(true, true) -> false
setAlarm(false, true) -> false
setAlarm(false, false) -> false
setAlarm(true, false) -> true
```

Good luck!

***
_This [challenge](https://www.codewars.com/kata/568dcc3c7f12767a62000038) comes from Swolebrain on CodeWars. Thank you to [CodeWars](<https://codewars.com/>), who has licensed redistribution of this challenge under the [2-Clause BSD License](<https://opensource.org/licenses/BSD-2-Clause>)!_

_Want to propose a challenge idea for a future post?  Email **yo+challenge@dev.to** with your suggestions!_