teleren/devto huggingface dataset:
Web Scraping HN with Python
Tags: python, web
Published: 2019-07-16 10:14:33

---
title: Web Scraping HN with Python
published: true
description: python, web
tags: python, web
---
You may know <a href="https://news.ycombinator.com/">HN</a>. A news aggregator with tech articles.

Let's scrape that with Python. The PyQuery module allows you to query HTML pages. You can collect all the links with PyQuery
 
    #!/usr/bin/python3
    from pyquery import PyQuery as pq

    doc =pq(url = "https://news.ycombinator.com/front?day=2019-07-14" )
    
    for link in doc('a.storylink'):
        print(link.attrib['href'])

That returns the links for the day "2019-07-14". So you have a list of links printed to the screen. You want that in a file.

<img src="https://proxy.duckduckgo.com/iu/?u=https%3A%2F%2Fmedia.tenor.co%2Fimages%2Fb910eef482ebf7f15aa6da0cc648f827%2Fraw&f=1">

Hockey dockey.

You can save the output into a csv file. A csv file is a file with all values stored with a delimiter in between, usually a colon but we'll use a semicolon.

    #!/usr/bin/python3
    from pyquery import PyQuery as pq

    date = "2019-07-14"
    doc =pq(url = "https://news.ycombinator.com/front?day=" + date )

    links = []
    for link in doc('a.storylink'):
        links.append(link.attrib['href'])

    with open('output.csv','w+') as csvfile:
        for link in links:
            csvfile.write( date + ";" + link + ";" )
            csvfile.write('\n')

Simple right? :) Run it and you'll have all the links in a nicely formatted csv file.

A csv file can read with an office program (any spreadsheet) or you can read them using <a href="https://pythonprogramminglanguage.com/read-csv/">Python pandas</a>.

Related links:
* <a href="https://pythonbasics.org/">Learn Python Programming</a>
* <a href="https://pythonbasics.org/exercises/">Python Exercises</a>
Dev-Log #4: Lander-game. Putting it on hiatus.
Tags: elixir, game, gamedev, phoenix
Published: 2019-07-16 10:06:38

My little lander-game will be put on hold for some time.

I like to keep my plans and ideas small and short, but I can't figure out how to make it fun in the little time I have available.

![](https://thepracticaldev.s3.amazonaws.com/i/g0lqbogehamk2n8cyh7h.jpg)

The main idea is that the player is assigning incoming space-ships to landing-pads. There are 5 pads. Every ship is owned by one of three fractions (police, corporate, mob). Those fractions like, dislike or are neutral to each other. When you place two ships from disliking fractions next to each other, your reputation with that fraction suffers, and vice versa. Better reputation means more money from the ships, which you need to keep the port running.

The latest idea was to leave it to the player to find out, which fraction likes/dislikes which other fraction. The other variant is to introduce those relationships one by one to the player.

There would also be further effects and events, like losing the game if every platform is occupied by mob-ships. Or getting extra money from the mob if there are no police-ships in the port.

All of this already works in the game, but it's still not fun to play. It feels like work, and I don't really enjoy playing it.

It was still a great way to get my feet wet with Elixir and Phoenix, and if anyone would like to see the code, or try the game, please comment! 

(It's in private repos on GH and it consist of the core-game, a CLI for the game, and the Phoenix-Web-App (no JS))

![](https://thepracticaldev.s3.amazonaws.com/i/1n9bpzfelfimxewvw1c2.png)

Holler if you like the idea and/or have other ideas how to make it more fun. Also feel free to steal this idea if you like.
Sorting with Life
Tags: algorithms, beginners, life, discuss
Published: 2019-07-16 09:45:57

---
title: Sorting with Life
published: true
description: Quick Sort
tags: #Algorithms #beginners #life #discuss
---
We often come across situations where we start blaming someone for something without being bothered why he failed to please us. And why not because it is quite obvious to point that very finger.
At that moment that person seems obnoxious to us. Later if some one is lucky enough then that point is taken back and pondered. And things gets sorted as an array. I sense a similarity here with Bubble sort. Bigger and good things are sent in the last and there is a focus on small elements. In bubble sort, an array of elements are divided into two segments: unsorted and sorted. The sorted part starts building at the end. We keep arguing every time from the start and drop the bigger(read better) parts at the end. 
```
begin BubbleSort(list)
   for all elements of list
      if list[i] > list[i+1]
         swap(list[i], list[i+1])
      end if
   end for
   return list
end BubbleSort
```
Thus the time complexity of that issue increases heavily. Good thing is, it will be sorted but you will have to pay the price with the time.

Let's take a step back and think if we can apply our favorite quick sort algorithm.
```
Quicksort(A,p,r) {
    if (p < r) {
       q <- Partition(A,p,r)
       Quicksort(A,p,q)
       Quicksort(A,q+1,r)
    }
}



Partition(A,p,r)
    x <- A[p]
    i <- p-1
    j <- r+1
    while (True) {
        repeat
            j <- j-1
        until (A[j] <= x)
        repeat
            i <- i+1
        until (A[i] >= x)
        if (i A[j]
        else 
            return(j)
    }
}

```

Now thing is why can't we take this step before rather than making things worse and delayed. Yes its true that sometimes arguments results into better understanding. But there exists a word "Sometimes".


Can a cautionary step taken at that time can avoid these mishaps?
In that short span of time when effervescence of anger fumes at peak is it possible to fit in others shoes and THINK. Well our dear Pivot element can serve the purpose here. Quick sort algorithm smartly selects the pivot element more lesser time it takes to sort. 
Can we intelligently select that pivot and divide the issue?
Well there lies an ambiguity and that's why we have best case, average case and the worst case. 


Let's conclude here with the person who fits in right there perfectly wins the moment and lives it. As the pivot was selected smartly. And this is why quick sort performs better than bubble sort because it initiates good processes from the starting. It doesn't wait long to start. 

## References
1. [Quick sort](https://runestone.academy/runestone/books/published/pythonds/SortSearch/TheQuickSort.html)
Top 10 Web Application Development Companies to Hire in 2019
Tags: webapplicationdevelopment
Published: 2019-07-16 09:44:03

Every business is going digital for improving their efficiency to make themselves visible across the world. Here's the list of <a title="top 10 web application development" href="https://yourstory.com/mystory/top-10-web-application-development-companies-to-hi">top 10 web application development</a> companies in the world.
Web scraping with PyQuery
Tags: python, web, scraping
Published: 2019-07-16 09:41:06

---
title: Web scraping with PyQuery
published: true
description: python
tags: python, web, scraping
cover_image: https://media1.tenor.com/images/0a476818e24f522d44f3d96cfb9d49d4/tenor.gif
---
How do you do webscraping these days? You may be working with <a href="https://pypi.org/project/beautifulsoup4/">beautifulsoup</a> or <a href="https://gumroad.com/l/Gjujxo">automate the web browser with selenium</a>.

If you have very basic scraping needs, you could consider pyquery.

pyquery allows you to make jquery *queries on xml documents*. That's great, because you can use it on HTML.

First intsall PyQuery with pip. Then you can use it like this:

    #!/usr/bin/python3
    from pyquery import PyQuery as pq

    doc =pq(url = "https://pythonbasics.org")
    print( doc('title').text() )

That will grab the title from the web page.
Want to get all links from a web page?

    #!/usr/bin/python3
    from pyquery import PyQuery as pq

    doc =pq(url = "https://dev.to")

    for link in doc('a'):
        print(link.attrib['href'])

Easy right?

Do you prefer getting images?

    #!/usr/bin/python3
    from pyquery import PyQuery as pq

    doc =pq(url = "https://dev.to")

    for link in doc('img'):
        print(link.attrib['src'])

Related links:

* <a href="https://pythonbasics.org/">Learn Python programming</a>
* <a href="https://pythonbasics.org/exercises/">Python Exercises</a>
Is anyone else using hCaptcha? It seems to work fine and got my first payout
Tags: hcaptcha, captcha, recaptcha, wordpress
Published: 2019-07-16 09:39:13

hCaptcha is basically reCaptcha which you can apply to your site and earn for visitors solving it. I run a forum and so far managed to cash out after hitting the minimum required. (used wordpress plugin) https://link.medium.com/k1KTaRzLmY

From some of their content i found this: Labor is placed into an anonymous open market for bidding. Companies all over the world bid for that labor to complete simple tasks easy for people but hard for machines. Websites earn revenue from that work instead of donating it to Google. Users support the site they’re visiting, and experience less web spam.

https://hcaptcha.com

So AI/ML companies are the ones who are paying for labeling services so at least there's a market being made here. Also looks like they're using ETH's blockchain to help this work altogether with reputation oracles to check the quality of those solving captcha, so you'd earn based on the quality of visitors (bots vs humans). It should be around 1000 solves for $1? 

Also, been seeing some concerning news over reCaptcha basically being a "Google Pixel". Quite concerning but i don't know to what extent they can extract from the mass of info they collect.

https://www.fastcompany.com/90369697/googles-new-recaptcha-has-a-dark-side

https://www.digitalinformationworld.com/2019/07/googles-new-recaptcha-tracks-you.html

I'm curious if other Devs know about this and if there are other alternatives?
Typescene Framework: Quick Start
Published: 2019-07-16 09:11:02

I've spent 5 years writing a JavaScript framework on my own.

Typescene is a robust front end library written in TypeScript: strongly typed, no dependencies, no nonsense. It’s really great for desktop-like (or mobile) applications, not so great for blogs and other content. It isn’t backed by some major corporation, not even a startup, but it’s been built by me: one developer on a mission to build a no-nonsense dependency-less framework.

[Original article](https://medium.com/@jcormont/ive-spent-5-years-writing-a-javascript-framework-on-my-own-af1201f4075c)

New article! Quick start, where I explain how Typescene works using a very practical example: a simple feedback widget.

[Typescene Framework: Quick Start](https://medium.com/@jcormont/typescene-framework-quick-start-f21c87b1f64d)
ScriptProperties Gotcha in Google Apps Script
Tags: googleappsscript, javascript, googlesheets, googledocs
Published: 2019-07-16 08:19:38

---
title: ScriptProperties Gotcha in Google Apps Script
published: true
description: Store a number, retrieve a string
tags: google-apps-script, javascript, google-sheets, google-docs
series: Google Apps Script
---
For reasons of insanity I have wrapped the `ScriptProperties` of the `PropertiesService` in a object with `get`, `set`, `forget` and `getKeys` methods, viz:
```javascript
function ScptProps() {
  this.scriptProperties = PropertiesService.getScriptProperties();
}

ScptProps.prototype.get = function (name) {
  return this.scriptProperties.getProperty(name);
};

ScptProps.prototype.set = function (name, value) {
  return this.scriptProperties.setProperty(name, value);
};

ScptProps.prototype.forget = function (name) {
  return this.scriptProperties.deleteProperty(name);
};

ScptProps.prototype.getKeys = function () {
  return this.scriptProperties.getKeys();
};
```
Using the REPL from my previous posting, I issued the following commands:
```javascript
(new ScptProps).set('goose',58);
typeof (new ScptProps).get('goose');
(new ScptProps).forget('goose');
```
Goose is me and 58 my age for those interested. 

And the gotcha? Well, I was a little taken aback recently, while debugging a number to number comparison issue, to discover that when I store a number I don't get one back. I get a string back and have to do a `parseInt()` on it to get its original value. The result of `typeof (new ScptProps).get('goose');` is, you guessed it, `string`!
Yarn workspaces - splitting React app by routes
Tags: help, discuss, react, yarn
Published: 2019-07-16 08:25:25

---
title: Yarn workspaces - splitting React app by routes
published: true
tags: help, discuss, react, yarn
---

Hello developers! 
I need your suggestions/help regarding the idea we had in our team. 

We have a massive React/Redux/Sagas application which weights a lot, so we decided to split it based on routes. But suddenly one idea came to our mind: why not to move those "split routes" code into separate yarn packages (with reducers, sagas, etc)? We thought it would be easier to manage in the future.

Splitting code into multiple chunks that would be imported dynamically with Webpack also requires us to implement additional logic for importing reducers and sagas after the chunk is downloaded by the browser, but it's doable.

Could you please share your thoughts on this? Does it really make sense to go this way and use yarn workspaces?

Thanks!
The Ultimate List of PHP Conferences
Tags: conferences, php, developer
Published: 2019-07-16 08:04:07

---
title: The Ultimate List of PHP Conferences
published: true
description: PHP conferences is where you can meet other developers and learn about PHP and web tools and best practices from the best developers in the community. Below is my ultimate list of the greatest PHP conferences in Europe and USA.

tags: conferences, php, developer

---

PHP conferences is where you can meet other developers and learn about PHP and web tools and best practices from the best developers in the community. Below is my ultimate list of the greatest PHP conferences in Europe and USA.


---

## DevTernity (🏅 Top 3 in Europe)

**Audience**: developers, architects ·  **Location:** Riga, Latvia · **Date**: 3–4 December 2020

[![Conference](https://avatars.io/twitter/devternity)](https://devternity.com)

**[DevTernity](https://devternity.com)** is the top 3 international software development conference in Europe. Located in the heart of Europe, in a beautiful city of Riga, this well-organized conference focuses on the core skills paramount to your success – code design, software architecture and leadership. Organizers promise no agile bulsh\*\*t or marketing talks. **This is a must-see event!**


**[See the full list](https://sizovs.net/conferences/php/)**
The Ultimate List of JS and Web Conferences
Tags: conferences, javascript, web, developer
Published: 2019-07-16 08:03:11

---
title: The Ultimate List of JS and Web Conferences
published: true
description: JS and Web conferences is where you can meet other developers and learn about web tools and best practices from the best developers in the community. Below is my ultimate list of the greatest JS and web conferences in Europe and USA.

tags: conferences, javascript, web, developer

---

JS and Web conferences is where you can meet other developers and learn about web tools and best practices from the best developers in the community. Below is my ultimate list of the greatest JS and web conferences in Europe and USA.


---

## DevTernity (🏅 Top 3 in Europe)

**Audience**: web developers, architects ·  **Location:** Riga, Latvia · **Date**: 3–4 December 2020

[![Conference](https://avatars.io/twitter/devternity)](https://devternity.com)

**[DevTernity](https://devternity.com)** is the top 3 international software development conference in Europe. Located in the heart of Europe, in a beautiful city of Riga, this well-organized conference focuses on the core skills paramount to your success – code design, software architecture and leadership. Organizers promise no agile bulsh\*\*t or marketing talks. **This is a must-see event!**


**[See the full list](https://sizovs.net/conferences/javascript-and-web/)**
Daily Challenge #15 - Stop gninnipS My sdroW!
Tags: challenge
Published: 2019-07-15 21:18:54

---
title: Daily Challenge #15 - Stop gninnipS My sdroW!
published: true
series: Daily Challenge
description: Our fifteenth daily challenge! Week 3, Day 1 - easy/medium difficulty
tags: challenge
---

Good morning, everyone. 
Let's warm up for the start of the week by playing with strings.

s'yadoT egnellahc semoc from user xDranik on sraWedoC!

> Write a function that takes in a string of one or more words and returns the same string, but with all words with five letters or more reversed. Strings passed in will consist of only letters and spaces.

Be sure to keep the order of the words the same and only reverse the letters.

Good luck, yppaH gniedoc!
***
_Thank you to [CodeWars](<https://codewars.com/>), who has licensed redistribution of this challenge under the [2-Clause BSD License](<https://opensource.org/licenses/BSD-2-Clause>)!_

_Want to propose a challenge for a future post?  Email **yo+challenge@dev.to** with your suggestions!_
node-gyp support in alpine linux
Tags: docker, devops, alpine, linux
Published: 2019-07-16 07:22:26

---
title: node-gyp support in alpine linux
published: true
description: Dockerize NodeJS applications using Alpine Linux with ease
tags: docker, devops, alpine, linux
cover_image: https://nickjanetakis.com/assets/blog/cards/the-3-biggest-wins-when-using-alpine-as-a-base-docker-image-3a9e4c1ccd2c8d88c771f35c6b96d47329e33f5cd969373e39fd610e3ced66df.jpg
---

Originally published on [Medium](https://medium.com/devgorilla/node-gyp-support-in-alpine-linux-9374c6191140)

When using alpine, you need to install build dependencies for some node module to be able to be built natively.
Here is an example of how you would install dependencies for packages that require node-gyp support on the alpine variant:
```
FROM node:alpine

RUN apk add --no-cache --virtual .gyp python make g++ \
    && npm install [ your npm dependencies here ] \
    && apk del .gyp
```
And Here’s a multistage build example:
```
FROM node:alpine as builder

## Install build toolchain, install node deps and compile native add-ons
RUN apk add --no-cache python make g++
RUN npm install [ your npm dependencies here ]

FROM node:alpine as app

## Copy built node modules and binaries without including the toolchain
COPY --from=builder node_modules .
```
## Pro Tip:
As Alpine Linux uses `musl`, you may run into some issues with environments expecting `glibc`-like behavior — especially if you try to use binaries compiled with `glibc`. You should recompile these binaries to use `musl` (compiling on Alpine is probably the easiest way to do this).
If you get an error similar to error loading shared library `ld-linux-x86-64.so.2`, it may be that you have dependencies relying on `libc` – you can try to fix this by adding:
```
RUN apk add --no-cache libc6-compat
```
 or 
```
RUN ln -s /lib/libc.musl-x86_64.so.1 /lib/ld-linux-x86-64.so.2
```
to your Dockerfile.

## Like to learn?
Follow me on [twitter](https://twitter.com/Grigortw) where I post all about the latest and greatest JavaScript, AI, DevOps, VR/AR, Technology, and Science! Connect with me on [LinkedIn](https://www.linkedin.com/in/grigorkh/) too!
Flatlist is not re rendering on props update
Tags: reactnative, redux
Published: 2019-07-16 06:38:45

I was trying to update an icon in flatlist according props. But its seems though the props is updated, no changes is reflected in view. I consoled data in every flow that is going on , to check what's missing. Roaming around the console for a while resorted to stackOverflow and found the solution https://stackoverflow.com/a/51416714/3237884 .
Rust async await (2)
Tags: rust, async
Published: 2019-07-16 06:21:33

---
title: Rust async await (2)
published: true
description: Rust async await
tags: rust, async
---

[Rust async await (1)](https://dev.to/x1957/rust-async-await-2l4c) 里面我们提到了简单的async await目前的语法，想用于实战试试。

我们用异步http请求试试，依赖如下:
```toml
[dependencies]
tokio = "0.1.22"
futures-preview = { version = "=0.3.0-alpha.17", features = ["compat"] }
reqwest = "0.9.18"
```

```rust
#![feature(async_await)]

use futures::compat::Future01CompatExt;
use futures::compat::Stream01CompatExt;
use futures::future::{FutureExt, TryFutureExt};
use futures::stream::{StreamExt, TryStreamExt};
use futures::Future;
use reqwest::r#async::{Client, ClientBuilder, Decoder};
use std::io::{self, Write};

async fn download() {
    let client = ClientBuilder::new().build().unwrap();
    let res = client
        .get("https://dev.to/x1957/rust-async-await-2l4c")
        .send()
        .compat()
        .await;
    let mut body = res.unwrap().into_body().compat();
      while let Some(next) = body.next().await {
        let chunk = next.unwrap();
        io::stdout().write_all(&chunk).unwrap();
    }
    println!("\nDone");
}

fn main() {
    let fut = download().unit_error().boxed().compat();
    tokio::run(fut);
}
```

需要注意几点：
1、*use reqwest::r#async::{Client, ClientBuilder, Decoder}* 因为reqwest还没有升级，之前的async现在已经是关键字了，所以得用r#
2、.send之后的future是futures01需要compat
3、into_body之后的stream是stream01需要compat
Is PHP dead? What is the job outlook and future of PHP in the next five years?
Tags: php
Published: 2019-07-16 06:18:24

PHP is far from dead. Since we get similar questions all the time, I am just going to repeat an answer from another discussion.

Is PHP still relevant as of 2019?

COBOL, a 1959 dinosaur language deployed on few tens of thousands installations is still relevant in 2019.

PHP is actively developed on 2019 and deployed on several hundreds of millions installations.

Therefore even just the “still widely used in many web apps” is plenty enough to justify learning it.

Plus, PHP is so easy to learn and used everywhere that not knowing it is:

    a bad reference on your curriculum,
    like not knowing any shell or bash command. Sure, you can ignore those but it shows the moment you are assigned some simple tasks. Your boss will know. Your colleagues will know.

Is PHP good for anything?

PHP, especially after version 7.0, is fast, simple, very syntax rich. It comes with embedded security libraries while being the cheapest language available on every host provider.
PHP runs on multiple CPU cores (Node.js needs additional work) and can be used both as sync and async (like Node.js) with an additional script.

PHP implements type hinting, interfaces, facades, traits, singletons, closures, structured exceptions, generic containers, iterators, namespaces, packages (Composer, akin to npm), unit tests, continuous integration…
It comes with thousands of packaged (and stand-alone) libraries, MVC / REST frameworks. Some, in example Laravel, are an industry standard.
And all of this can be learned one bit at a time, because basic PHP is as simple as writing “print(“Hello world”);”. Even a child can learn PHP.
As such, you can find cheap PHP developers everywhere in the world, not just in California.
click here to continue reading: https://simpliv.wordpress.com/2019/07/16/is-php-dead-what-is-the-job-outlook-and-future-of-php-in-the-next-five-years/
Do you include hobbies in your CV?
Tags: discuss, cv
Published: 2019-07-16 06:07:11

---
title: Do you include hobbies in your CV?
published: true
description: 
tags: discussion, cv
---

A friend asked for feedback on their CV, and I thought adding some hobbies is a good idea, as it gives a little more depth as a person.
He strongly disagreed, saying the proper place to mention hobbies is F2F only.

What do you think?
Do you include hobbies in your CV?