teleren/devto huggingface dataset:
Systematic Web Scraping

Published: 2020-09-30 17:22:18

If it helps to think of a web crawler as a system than a piece of code.

This shift is very important and will be forced on any developer whoever attempts web scraping at scale.

It is one of the best ways to learn thinking in systems.

We can see the whole crawling process as a workflow with multiple possible points of failure. in fact, any place where the scraper is dependant on external resources is a place it could and will fail. So 90% of the time spent by the developer is in fixing in bit and pieces these inevitable issues.

At Proxies API, we have gone through the drudgery of not thinking in a systematic way about web scraping till we one day took a step back and identified the central problem. The code was never the problem. The whole thing didn't work as a system. We finally decided on our own set of rules to make crawlers that work systematically.

Here are the rules that the system has to obey:

Handle fetching issues (timeouts, redirects, headers, browser spoofing, CAPTCHAs and IP blocks)
Where the crawler doesn't have a solution to each of the issues(for example, CAPTCHAs), it should at least handle and log them.
The system should be able to "step over" any issue and not stumble and fail to bring everything down with it.
The system should immediately alert the developer about an issue.
The system should help the developer diagnose the last issue quickly with as much context as possible, so it is easily re-producable
The system should be as generic as possible at the code level and should push individual website logic to an external database as much as possible.
The system should have enough levers to control the speed and scale of the crawl.
Basics of ELK Stack(Elasticsearch-Logstash-kibana) Part -01 
Published: 2020-09-30 17:21:17

**What is ELK Stack?**

-Collection of three open source products
 1.ElasticSearch - used for storing logs
 2.Logstash - used for shipping as well as processing and storing 
              logs
 3.Kibana - visualization tool

**Motive behind ELk**
-Designed to take data from any source, in any format and to search ,analyze, and visualize that data in real-time.
-Provide centralized logging that be useful when attempting to identify problems with servers or applications.


**Simple Architecture For ELK Stack**

Log File -> Logstash(Data Processing) -> ElasticSearch(Storage) -> Kibana(Visualize)

**Logs** - Server Logs that need to be analyzed.
**Logstash** - Collect logs and events data and parses and 
               transfers data to ElasticSearch.
**Elasticsearch** - The transformed data from logstash is 
                   store,search and indexed.
**Kibana** - Kibana Uses ElasticSearch DB to Explore ,Visualize 
            and Share.
📊 New JS Chart Features in AnyChart 8.9.0
Tags: javascript, showdev, dataviz, datascience
Published: 2020-09-30 17:20:32

[AnyChart 8.9.0](https://www.anychart.com/blog/2020/09/29/new-javascript-chart-features/) is out. The latest major update brings new cool out-of-the-box features to this entire family of JavaScript charting libraries — AnyChart, AnyStock, AnyGantt, and AnyMap. Try the improved functionality to build even more compelling data visualizations in less time.

Quick highlights:

💥 Label text length limit
💥 Client-side CSV/XLSX export
💥 Finite trend channel annotation
💥 Volume-weighted average price indicator
💥 Gantt chart labels overlap
💥 Improved placement of the first and last labels
💥 Waterfall stack labels (multi-series)
💥 Automatic placement of polar points

[Learn more about AnyChart 8.9.0](https://www.anychart.com/blog/2020/09/29/new-javascript-chart-features/) on the AnyChart blog.
Multiple usage of custom filters in Vue.js
Tags: vue, filters, bestpractices
Published: 2020-09-30 17:18:44

---
title: Multiple usage of custom filters in Vue.js
published: true
description: Best way to use custom filters in Vue.js
tags: vuejs, filters, bestpractices
//cover_image: https://dev-to-uploads.s3.amazonaws.com/i/2zdpyun3esbzseyful7k.png
---

A Vue.js filter is simply a function that takes an input, processes it and gives an output. Inside a Vue template it is written after a single pipe and can also be followed by its arguments.

Example of Vue.js filter

```javascript
<span v-html="timestamp | formatDate"></span>
<span> Also like this: {{timestamp | formatDate}} </span>
```

But there can be some particular case where the filter functions are required to be used outside directive and mustaches. This article will show a simple mechanism to tackle the problem.

---

A custom filter can be registered either globally or inside components. Since we are doing it the best way, we will be registering it globally.
If you have many filters, its best to keep it on a separate folder inside separate files. For this particular case, I have kept it inside `/src/filters.js`

![Folder Structure](https://dev-to-uploads.s3.amazonaws.com/i/usp6g09uilsov3te9cyo.png)
 
Now lets take a look inside `filters.js` file.

```javascript
import Vue from 'vue'
import moment from 'moment';

export const FILTERS = {
    formatDate: (value) => {
        return value ? moment(value, "x").format("DD MMM YYYY") : '';
    },

    ellipsis: (value, MAX_LEN) => {
        if (value) return value.length > MAX_LEN ? (value.slice(0, MAX_LEN) + '...') : value;
        return null;
    },
}

export const registerFilters = function () {
    Object.keys(FILTERS).forEach(key => {
        Vue.filter(key, FILTERS[key]);
    })
}
```

As you may have noticed, the `FILTERS` object contains all of your filter functions. The key serves as `filterId` and the value as `filter function`

The file exports two things
- `FILTERS` object
- `registerFilters` function

That’s the main point. The `registerFilters` function registers all of the filter functions globally so that it can be used on any Vue component. The method is called inside the `main.js` file.

```javascript
// Filters
import { registerFilters } from "./filters";
registerFilters();
```

Another cool thing about this approach is the ability to import those filter functions and use inside JavaScript logic.

```javascript
<script>
	import { FILTERS } from "@/filters";

	export default{
		name: 'TestComponent',
	
		methods:{
			testMethod(){
				let date = new Date();
				return FILTERS.formatDate(date);
			}
		}
	}
</script>
```

In many cases filter functions need to be used outside of directive and mustaches. This approach works best to handle this particular scenario. Moreover, it groups the filter functions at one place and makes a clear separation of logic inside vue application.

Happy Coding !
Jualan Tanpa Modal Ide Bisnis Modern
Published: 2020-09-30 16:52:47

FAQs https://www.putriadisty.my.id

Pertanyaan: Apa Yang Dimaksud Tanpa Modal?
Jawaban: Tanpa Modal disini maksudnya adalah, anda tidak perlu mengeluarkan uang untuk biaya produksi. Sehingga anda hanya tinggal promosi jualan, dan mendapatkan keuntungan dari situ.

Pertanyaan: Apakah Betul Tanpa Modal Sepeser pun?
Jawaban: Saya anggap, pulsa internet dan termasuk handphone atau laptop bukan kedalam modal. Karena seharusnya setiap pelaku usaha/bisnis sudah mempersiapkan beberapa hal tersebut.

Pertanyaan: Apa Saja Bentuk Bisnis Tanpa Modal Itu?
Jawaban: Dropshipper, Reseller, dan Affiliasi.

Pertanyaan: Bagaimana Saya Bisa Memulai nya?
Jawaban: Silahkan baca selengkapnya disini.
JS 🔥: Conditionally setting an object property
Tags: javascript, webdev, beginners, tutorial
Published: 2020-09-30 16:36:14

Want to get better at Web Development 🚀🚀🚀? Subscribe to my weekly newsletter at **[https://codesnacks.net/subscribe/](https://codesnacks.net/subscribe/)**

---

Let's assume we only want to add an object property if some condition is true. We can do this using an if-statement of course:

```js
const someCondition = true;
const anotherCondition = false;
const myObject = {
  name: "codesnacks",
};

if(someCondition){
  myObject.author = "Ben";
}

if(anotherCondition){
  myObject.platform = "dev.to";
}

console.log(myObject); // {name: "codesnacks", author: "Ben"}
```

We can achieve the same using the object spread operator ( `...`) in combination with the condition when creating the object. No additional if-statement is needed. This is especially elegant if an object has multiple conditional properties.

```js
const someCondition = true;
const anotherCondition = false;
const myObject = {
  name: "codesnacks",
  ...(someCondition && { author: "Ben" }),
  ...(anotherCondition && { platform: "dev.to" }),
};

console.log(myObject); // {name: "codesnacks", author: "Ben"}
```

---

Want to get better at Web Development?
[🚀🚀🚀subscribe to the Tutorial Tuesday ✉️newsletter](https://codesnacks.net/subscribe/)
🔥 How to Handle with Multiple Windows,Frames and Iframe in Selenium(with Code) 
Tags: testing, selenium, webdev, beginners
Published: 2020-09-30 16:35:04

✅ Join us - https://sendfox.com/thetestingacademy

✅ Download Code - https://codeshare.io/5O1nOW

{% youtube 7np7iWZMXgc %}

In this video, We are going to learn How To Handle Handle with Multiple Windows,Frames and Iframe in Selenium.

✅ API Testing using Postman -
https://www.learnapitesting.com

✅ Cypress Tutorial with LIVE Projects -
https://cypresstutorial.com

✅ REST API testing with Python -
https://masterapitesting.com

✅ Handling Multiple Windows in Selenium
The user scenario being automated here is: Open a new tab and then switch back to the last window to complete the other pending activities. In such scenarios, Selenium helps to handle multiple windows through window handlers and javascript executors.

✅ What is a window handle?
It is a unique identifier that holds the address of all the windows. Think of it as a pointer to a window, which returns the string value. It is assumed that each browser will have a unique window handle. This window handle function helps to retrieve the handles of all windows.

✅ Syntax

get.windowhandle(): This method helps to get the window handle of the current window
get.windowhandles(): This method helps to get the handles of all the windows opened
set: This method helps to set the window handles in the form of a string. set string  set= driver.get.windowhandles()
switch to: This method helps to switch between the windows
action: This method helps to perform certain actions on the windows
These are some of the methods that will be used to handle multiple windows in Selenium.

🚀 What is 30 Days of Automation in Testing Challenge?

This video is part of 30 Days of Automation in Testing Challenge in Selenium where we will create our own Automation Framework from start. 

 #seleniumtutorial #thetestingacademy #seleniumframework #seleniumjava
#seleniumforbeginner
Welcome Thread - v93
Tags: welcome
Published: 2020-09-30 16:18:20

---
title: Welcome Thread - v93
published: true
description: Welcome to DEV! A thread of hellos and intros.
tags: welcome
canonical_url: 
---

![waving animals in masks](https://media.giphy.com/media/kaIJGG8lNC3u5DM3tg/giphy.gif)

### Welcome to DEV!

1. Leave a comment below to introduce yourself! You can talk about what brought you here, what you're learning, or just a fun fact about yourself.

2. Reply to someone's comment, either with a question or just a hello. 👋 

**Great to have you in the community!**
Using JavaScript Proxies
Tags: javascript, webdev, es6
Published: 2020-09-30 16:12:33

In this post I will show you how to use a **Proxy**.

The concept of a proxy is pretty simple. You provide triggers that run when you are getting or setting something in an object.

```javascript
// private user object
const _user = {
	name: "User",
	age: 25,
	_address: "A place"
}

const traps = {
	get(target, prop) {
		if (prop[0] === "_") {
			return undefined;
		}
	}
}

const user = new Proxy(_user, traps);

console.log(user.name) // User
console.log(user.age) // 25
console.log(user._address) // undefined
```
As shown in the example above, Proxies can be used for stopping programs from accessing private variables.
They can also be used to stop programs from setting private variables.
```javascript
// private user object
const _user = {
	name: "User",
	age: 25,
	_address: "A place"
}

const traps = {
	get(target, prop) {
		if (prop[0] === "_") {
			return undefined;
		}
	},
	set(target, prop, value) {
		if (prop[0] === "_") {
			return;
		} else {
			target[prop] = value;
		}
	}
}

const user = new Proxy(_user, traps);

user.name = "Person";
user.age = 26;

user._address = "In the world"; // Doesn't set
```

You can read more about **Proxies** on the [MDN](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Proxy).
Day 16 - Make A Simple Animated Nav Button 
Tags: codenewbie, beginners, css
Published: 2020-09-30 16:04:30

I have been enjoying making animated navigation buttons - the ones that change from hamburger lines to an X when the navigation bar is opened. This time around, as I update my portfolio, I wanted to build one from scratch instead of reusing my previous buttons.

![example gif](https://media.giphy.com/media/EukDSfW5oQPIqGF2fp/giphy.gif)

The one I made is in the shape of a circle. First, I made the button container and the hamburger lines. 

![Set Up](https://dev-to-uploads.s3.amazonaws.com/i/lh76ncx585q4zfacxapt.png)
<figcaption> html set up </figcaption>

![css set up](https://dev-to-uploads.s3.amazonaws.com/i/6qfrssmsd0j80g0ajbyi.png)
<figcaption> CSS set up <figcaption>

I also created a class for CSS flex to align everything inside the hamburger wrapper vertically and horizontally.

![flex center](https://dev-to-uploads.s3.amazonaws.com/i/5f9sng6vvgwg4brayilj.png)

Since the hamburger line class was already assigned to the three divs in the html file, we should see the following:

![example starting point](https://dev-to-uploads.s3.amazonaws.com/i/l0s0f6zqswqhlkgsba3h.png)

I chose the following transformations to make the X: 

- Top line will rotate 45 degrees and move down 10px
- Middle line will disappear
- Bottom line will rotate the opposite way -45 degrees and move up 10px

![line classes](https://dev-to-uploads.s3.amazonaws.com/i/t2844ex8yyhjdc42fujo.png)

In this example, the animations are done by using the CSS transition property. So I added a class for the transition property and assigned it to our hamburger lines: 

![transition property](https://dev-to-uploads.s3.amazonaws.com/i/dhs8mj8102x2txvua6jm.png)

![assigned transition](https://dev-to-uploads.s3.amazonaws.com/i/73codueodajx7cauuy04.png)

Unique identifiers were set for the top, middle, lines in order to select them with Javascript. In this one I used ids.

![ids example](https://dev-to-uploads.s3.amazonaws.com/i/tarfixvj1q8zdjxqpp66.png)

Almost done! Using Javascript, I can dynamically toggle the classes ending with "-animate" so the transition property smoothly changes the lines to create an X.

The Javascript code should do the following:

- Find the button element in the html document
- Create a function that toggles the "-animate" class names in the html elements
- Create an event listener on the button element that calls the function to activate the transition property every time the button is clicked

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/lvhngbcvn5g24o414rk4.png)

Check out the entire code [here](https://codepen.io/josno/pen/jOqoXKX).

## What I Learned
I was able to practice building navigation buttons from my memory!

## How This Will Help Me
I can get more creative with interactive navigation buttons!
Delete expired documents automatically with MongoDB (TTL index)
Tags: mongodb, database
Published: 2020-09-30 15:57:02

Recently I needed to delete some documents that I save in MongoDB after some time. I can think of a few examples why we would want to delete data after some time:

- logs / events
- api keys / access tokens
- non-active users
- etc

We could do it by running a cronjob that deletes the data, delete the data every time we insert new data or any other solution.

Luckily for me, my wife told me that MongoDB already have that mechanism built-in.

## TTL index

TTL (Time-To-Live) indexes are special single-field indexes that MongoDB can use to automatically remove documents from a collection after a certain amount of time.

A background thread in mongod reads the values in the index and removes expired documents from the collection (usually every minute).

For example, to create a TTL index on the `lastModifiedDate` field of the `eventlog` collection, use the following operation in the mongo shell:

```jsx
db.eventlog.createIndex( { "lastModifiedDate": 1 }, { expireAfterSeconds: 3600 } )
```

- The indexed field must be [BSON date type](https://docs.mongodb.com/manual/reference/bson-types/#document-bson-type-date) or an array of BSON dates
- If the indexed field in a document is not a date or an array that holds a date value(s), the document will not expire.
- If a document does not contain the indexed field, the document will not expire.

## Conditional delete

As of MongoDB 3.2, a collection can be partially indexed using a specified filter expression, `partialFilterExpression`. TTL index can also be used with partial indexes.

For example:

Delete documents that created 1 hour ago and their `state` is `TMP`

```jsx
db.eventlog.createIndex(
  { created_at: 1 },
  { expireAfterSeconds: 3600, partialFilterExpression: { state: 'TMP' } }
);
```

Delete documents that created 1 day ago and their `count` is lower than `5`

```jsx
db.eventlog.createIndex(
  { created_at: 1 },
  { expireAfterSeconds: 86400, partialFilterExpression: { count: { $lt: 5 } } }
);
```

[Additional info on partial indexes](https://docs.mongodb.com/manual/core/index-partial/)

## Real world example

Recently I finished developing [BundleMon](https://github.com/LironEr/bundlemon), which is a free tool that helps you monitor your apps bundle size.

One of the components in BundleMon is a service that saves historic reports in order to compare bundle size between branches, So when you open a PR BundleMon saves a record with the current bundle size report.

No need to save the report more than 30 days, so I just added a TTL index:

```jsx
db.reports.createIndex(
  { creationDate: 1 },
  { expireAfterSeconds: 2592000, partialFilterExpression: { prNumber: { $exists: true } } }
);
```
16 vs 13-inch MacBook Pro for Node.js Development
Tags: discuss
Published: 2020-09-30 15:51:51

Hi everyone!
 
I’m using a MacBook Air (2013) with i5-4250U and want to update to some newer model. The main goal is to decrease server restart time of a big Node.js project and decrease time of TypeScript compilation. Currently it takes about 8 and 25 second respectively. As projects growing it starting really bothers me.
 
I’m choosing between:
- [13-inch 2020 with **4-cores i5-1038NG7**](https://www.apple.com/shop/buy-mac/macbook-pro/13-inch-space-gray-2.0ghz-quad-core-processor-with-turbo-boost-up-to-3.8ghz-512gb#)  
- [16-inch 2019 with **6 cores i7-9750H**](https://www.apple.com/shop/buy-mac/macbook-pro/16-inch-space-gray-2.6ghz-6-core-processor-512gb#)
 
**The question is**: is it worth to spend +600$ more for additional 2 cores? 
 
[The benchmarks](https://nanoreview.net/en/cpu-compare/intel-core-i7-9750h-vs-intel-core-i5-1038ng7) shows that the i7-9750H up to **35% faster** in multicore benchmarks, but in single-core the 1038NG7 can be even a bit faster. Also 9750H has **2 times bigger L3 cache**.
 
RAM and storage are the same, but the 13-inch model has faster memory: 3733MHz vs 2666MHz.
 
80% of the time I use an external monitor, so screen size is not a dealbreaker for me.
Ruby: Get and save google images with Selenium
Tags: ruby, beginners, webdev, rails
Published: 2020-09-30 15:51:05

---
title: Ruby: Get and save google images with Selenium
published: true
description: 
tags: ruby, beginners, webdev, rails
//cover_image: https://direct_url_to_image.jpg
---

Hi, here a little Ruby code to get and save images from Google Image. Can be handy :-)

If you need help with selenium web driver: https://help.crossbrowsertesting.com/selenium-testing/tutorials/getting-started-with-ruby-and-selenium/

```ruby
require 'selenium-webdriver'
require 'faraday'

def search_google(search_query)

  driver = Selenium::WebDriver.for :chrome
  search_url = "https://www.google.com/search?site=&tbm=isch&source=hp&biw=1873&bih=990&q=#{search_query}"
  images_url = []

  # Open the browser and navigate to Google search image
  #driver.navigate.to search_url
  driver.navigate.to search_url
  elements = driver.find_elements(class: 'rg_i')

  elements.each.with_index do |e, index|
    #Click on the image (to get the full size image)
    e.click
    sleep(1)
    
    #Get the full size image and put it's url in a array
    element = driver.find_elements(class: 'v4dQwb')

    #This is google site specific logic
    if index == 0
      big_img = element[0].find_element(class: 'n3VNCb')
    else
      big_img = element[1].find_element(class: 'n3VNCb')
    end
    images_url.push big_img.attribute("src")
    
    begin
      # Get images data 
      response = Faraday.get(images_url[index], {}, {})

      #Write image to search.jpg file
      File.open("search#{index+1}.jpg",'w') do |f|
            f.puts response.body
      end
    rescue => exception
      puts 'Error saving image'
    end  
    
    # will only read 5 images
    if index == 5
      break
    end

  end
end

search_google 'dog'
```

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/23ju31meq6j24w308fzx.jpg)
Refactoring Vue Templates in Nuxt
Tags: nuxt, vue, javascript, html
Published: 2020-09-30 15:50:52

Well, as soon as we start updating HTML in multiple places, it is time to start refactoring.

In order to make your HTML as reusable as possible, you should build components for all the parts of the HTML that you are using in multiple places.

With Nuxt, it is easy to do. 

To reuse HTML, you can build Vue components in the [components directory](https://nuxtjs.org/guides/directory-structure/components), and your components will automatically be registered whenever you use them in a page or another component.

**Here is one of my components for a page header:**
```vue
<template>
  <section>
      <div
        class="py-10 mx-auto max-w-screen-xl px-4 sm:py-12 sm:px-6 md:py-16 lg:py-20 xl:py-28"
      >
        <div class="text-center">
          <h1
            class="text-4xl tracking-tight leading-10 font-extrabold text-gray-900 sm:text-5xl sm:leading-none md:text-6xl"
          >
            {{ title }}
          </h1>
          <p
            class="mt-3 max-w-md mx-auto text-base text-gray-500 sm:text-lg md:mt-5 md:text-xl md:max-w-3xl"
          >
            {{ subtitle }}
          </p>
        </div>
      </div>
    </section>
</template>

<script>
export default {
  props: ['title', 'subtitle']
}
</script>
```

**And here is that component being used within a page template:**
```vue
<template>
  <div>
    <!-- This is the Component above -->
    <PageHeader :title='projectPage.title' :subtitle='projectPage.subtitle' />
    <!-- This is another Component I built -->
    <ProjectList :projects='projects' />
  </div>
</template>
```

As you may notice, I am using [vue props](https://vuejs.org/v2/guide/components-props.html) for the content so that I can use the same header on multiple pages, but only change the content. This will make maintenance and updates so much faster because I will only have to update the HTML in one place.

Now, you can see the live site at [chriswray.dev](https://chriswray.dev). Would love to know what you think!