teleren/devto huggingface dataset:
Divide RDD into sub parts
Tags: scala, spark
Published: 2019-08-31 11:22:40

<b>Problem Definition</b>

Divide the Spark RDD into N(as defined by user) sub parts and perform same function on these sub parts with different data set on each sub part

<b>Explanation</b>

I am having RDD. I want to divide that RDD into sub parts let call him partitions. Now on each partition i want to perform same task for multiple iteration.
Here is code that explains well

    case class shape ( dim:Int) {
      val random = new Random()
      var X      : Array[Double]       =   Array.fill(dim)(random.nextDouble() * (100-10)+11 )
      var Y      : Array[Double]       =   Array.fill(dim)( math.random)
      var Loss       : Double              =   math.random
      var NewLoss       : Double              =   math.random 
    }


    val N = 1000 //in real N will be in millions
    val d = 100 //in real N will be in millions
    val nP = 4  // nP defines number of sub parts into which RDD is divided
    val iterations = 1000 // in real analysis iteration will be in millions or billions  
    val list = List.fill(N)(new shape(d))

     list.map { x =>
      x.Loss = function. SphereFunc(x.X) // Update Loss of each element
    }
  
    val rdd = sc.parallelize(batList, nP)
    var partitioned = rdd.persist()
    
    for(iter <- 1 to iterations) {
    
     partitioned = partitioned.mapPartitionsWithIndex { (k, iterator  )  =>
     val li = iterator.toList
     val localBest = li.minBy(_.Loss).X
     li.map { j =>
            j.Y = ((j.X, localBest).zipped.map(_ - _).map(_ * math.random), j.Y).zipped.map(_ + _)
            j.X = (j.X, j.Y).zipped.map(_ + _)

           }
     li.filter(math.random > _.Loss)
            .map { j =>
              j.X = localBest.map(_ + math.random)            
           }
     li.map{j => j.NewLoss = SphereFunc(j.X)

     li.filter(j => math.random < j.NewLoss && j.NewLoss < j.Loss).map { j =>
            j.Loss = j.NewLoss  
           }
      li.iterator
    }

    def SphereFunc(list: List[Double]): Double = {
    list.foldLeft(0.0)((x, xs) => x + xs * xs)
    }


In this code I created RDD and for each iteration I call <b>mapPartitionsWithIndex</b> to get each partition data and updated elements of each partition.

Problem with this code is that for each iteration, when I call <b>mapPartitionsWithIndex</b> it will perform required operation on each element of partitions and it will again create RDD. So partitions will not remain same. For each iteration RDD is divided again to partitions. Also will cause lot of shuffeling.

But I want to create partitions from RDD at start and than perform desired operation on partitions till user defined iteration completes. 
Here i am calling <b>mapPartitionsWithIndex</b> iterations times. Is this possible to call <b>mapPartitionsWithIndex</b> only one time and than iterate over partitions till <b>iterations</b> 

How can I achieve that? I have to run experiments on cluster with more than 100 cores.
Don’t Hire Remote Workers Who Can’t Nail These Interview Questions
Published: 2019-08-31 10:58:31

Remote worker- The one who works outside the office or you can say that the one who works from home or from any remote location. A remote employee is employed by a company but works outside of a regular office cubicle.

Nowadays this is like a new convenient to work, that everyone wants to work remotely. It gives a proper explorer to the people work want to work in different countries and with different companies. Remote working gives you flexible working hours.

But is also important to clear the interview before being a part of the company. There are strict policy and rules to work as remote workers. Because remote working takes lots of responsibilities of the office as well as home. Being a remote worker is easy but working as a remote worker is quite difficult if you don’t know how to manage the time and your work efficiently. 

For more details about this article click here.
https://codersera.com/blog/dont-hire-remote-workers-who-cant-nail-these-interview-questions/
Building a WTF util from 0 to hero
Tags: go, tutorial, showdev
Published: 2019-08-31 10:34:49

Photo by Caspar Camille Rubin on [Unsplash](https://unsplash.com/photos/fPkvU7RDmCo)

# Episode 1 - Why the fuck (INTRO)

Not so long ago I discovered [wtfutil](http://wtfutil.com) an amazing developer oriented tool to monitor, check and browse a lot of stuff, it operates with modules and there is a module that I always find very useful called news.

This modules allows you to see a list of the latest [HackerNews](https://news.ycombinator.com/) articles and if you decide it is interesting enough, you can click on it and it will open the related article on your default browser, which for me is amazing because anyway I'm checking the website every day.

It is basically this daily routine what got me thinking, which other websites I'm checking often to be up to date with news around the tech world, here is where I had an Eureka moment and though **well if this thing is not out yet, lets create it.**

## The approach

First of all I love to share the few things I know, even if its not the best way of doing it I'll share it, and then when someone says hey this can be also done this way, I learn something new, and this is a golden cycle you should try, so for me this will be like a tutorial series on how to approach this from the initial project directory, till having the tool ready to be used by the wtfutil users.

So stay tuned.

Cheers :beer:

### Resources

{% github wtfutil/wtf %}

{% github VictorAvelar/devto-api-go %}
The joy of self answering
Tags: stackoverflow
Published: 2019-08-31 04:52:07

When Stack Overflow started, there wasn't a lot of content for me. Most people seemed interested in asking about obscure Microsoft products such as [C# and .NET](https://insights.stackoverflow.com/trends?tags=c%23%2C.net%2Casp.net%2Csql-server). I worked at a UNIX shop and we used reliable technology like [C and Perl](https://insights.stackoverflow.com/trends?tags=c%2Cshell%2Cperl%2Coracle). But I was intrigued by the idea of a site dedicated to questions and answers, so I thought I'd try my hand at asking questions.

By the way, Stack Overflow interested me because I had read [_Practical Internet Groupware_](http://shop.oreilly.com/product/9781565925373.do) by Jon Udell sometime in the early 2000s. (Or perhaps read _about_ the book enough to be excited about the concept.) The premise of the book was that the internet would radically change the way people interact with each other since it allows groups to collaborate at scale. It's impossible to explain how exciting this idea was since we now accept it as obviously true. But before Twitter, Facebook, podcasts, blogs or even forums, few people guessed that the internet would democratize access to public discourse.

Stack Overflow tackles one, very specific, problem: uneven access to arcane programming knowledge. It's hard for me to recall, but there was a time when every programmer had to discover on their own oddities such as:

{% stackoverflow 583889 %}

Then you'd need to walk around the office to see if anyone was proficient enough in shell commands to find a working solution. Or, just struggle through untarring hundreds of files by hand. (Ask me how I know this.) But with Stack Overflow, someone can come along, see the question and provide an answer:

{% stackoverflow 583891 %}

And then roughly 68 thousand people could find that very same answer and learn something about how the UNIX shell works.

The twist: it doesn't have to be different people who ask and answer. I figured out my own solution and instead of moving on with my life, I posted the answer to my own question. Maybe not the most monumental thing in the world, but rather my small way to help people out in the future.

Now it can be hard to ask a question as if you didn't know the answer and there are people on the site who are suspicious of self-answered questions. But if you wish there was a bit of programming knowledge available on the internet, you don't need to wait for someone else to ask the question.
Blockchain technology is something which can empower a wide range of processes in a variety of industries.
Tags: blockchaineducation, blockchainprogramming, blocklogy
Published: 2019-08-31 09:25:45

Blocklogy is an e-learning platform made to empower students (& young minds) with extensive knowledge and understanding of next-gen technologies like blockchain. The Blocklogy platform is designed to deliver career-oriented education with a focus on research & development. 

For more detailes:
https://www.blocklogy.org/blog/black-belt-expert-developments-part-2-a-level-in-blocklogy-explained/

Visit for more details: https://blocklogy.org


Subscribe to our Channel for more: https://www.youtube.com/watch?v=MJ7nfzVrxgw

For more updates follow us on:

Facebook: https://www.facebook.com/Blocklogy

Twitter: https://twitter.com/blocklogy

Linkedin: https://www.linkedin.com/company/blocklogy/

Instagram: https://www.instagram.com/blocklogyedutech/
Distinguished Devs #1: Carsten Haitzler - creator of Enlightenment
Tags: career, productivity, motivation, podcast
Published: 2019-08-31 08:59:34

Carsten Haitzler, AKA Rasterman, is the creator of Enlightenment, a window manager used in millions of Linux devices worldwide - from desktops and smartphones to smartwatches and TVs.

As the creator and leader of a large project for many years, in this interview he talks about how open source has changed and how he approaches leadership. Carsten also gives his thoughts on learning to code, and what makes a great developer.

{% soundcloud https://soundcloud.com/distinguished-devs/1-carsten-haitzler %}
The Distinguished Devs podcast is available on iTunes, Spotify etc.

I hope you enjoy this episode, please let me know what you think!

The next episode will be an interview with John Leider, creator of Vuetify,
so don't forget to hit follow.
Request for Comments: Slurping Kubernetes logs
Tags: help
Published: 2019-08-31 08:33:41

I've investigated many tools and stacks to store and search logs of containers, especially from Kubernetes clusters. In my environment, I have a hefty constraint that I don't want the logs, which main contain personal information, e.g. usernames in URIs, to be stored outside the EU. GDPR and all. That excludes the ready-to-use services like datadog, dynatrace, newrelic, etc.

It seems the [ELK stack](https://www.elastic.co/de/what-is/elk-stack) consisting of ElasticSearch, Logstash and Kibana seems pretty popular and can be used on-premise. However, investigating this and other solutions I found them all to be very complex, and read comments about them being hard to set up and maintain.

Thinking about it, I wonder whether there is not a very simple approach that might yield good results. My idea is simple: Just write a small script that fetches logs from containers via `kubectl logs --timestamps=true` once a minute or so. Consecutive fetches use  `--since-time` of the last received timestamp. The script pushes the results into some database to archive them and make them searchable.

I know, this is a "roll-your-own..." approach, but it seems pretty simple, but also effective. I'd like to ask the dev community for feedback on this idea, and pointers to alternatives that are not complex and can be used on-premise.

So, what are your thoughts?
IronHack BCN Challenge1, Code Part
Tags: replit, python
Published: 2019-08-31 07:55:11

---
title: IronHack BCN Challenge1, Code Part
published: true
tags: replit, python
---

Small code to process customer Id's, at least 30 Millions of ID's in 75 seconds on repl.it.
Stress Detailed test, and memory profiling optional.

{% replit @sfrias/Challenge1 %}

For fun, a problem that they insist on solving in python. list, minimum 28 million id's to process periodically. I start from a list of id's of 14 million dummy for tests (so it should go baggy in an instance in the cloud of 1GB of memory, and 1 vCPU, leftover storage, not SSD). As the list of id's that is being renewed periodically has to be accessible by other applications, and I don't want to succumb to the space and process with jSON, I do some tests with a simulated list of 14 million id's with facilities that allow the dump / recovery of data objects (pickle, marshal, HDF5 in tree B).

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/yhmy69m57oq5758gp95l.png)

The space required by the data structures is clear.They are extracted with getsizeof (), and match what is expected in python, including overhead. The list of 14M id's occupies 107MiB, which is 8 bytes per element, just like the 18M id's stress test in which the dump file occupies 138MiB (144006144 bytes). What is no longer so clear is why in the recovery of data objects, the memory is 5 times greater, choose the facility you choose. The evolution of memory is described below.

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/d885pwy17fbycb2x6sx2.png)

Finally I have made my own compatible reader / writer of arrays in python on disk, as a binary file. Anyway, it seems that it processes 30 million IDs in 75 seconds, which is already left over for the challenge with a single thread

These are the things that can happen when you use a tool with self-managed memory, in which there is a great diversity of packages, and in which the overhead object clearly sacrifices efficiency. There is a dilemma between changing tools or performing low-level operations yourself in some function, with cost in time and memory space known and stable. In this case we're sacrificing compatibility on read/write object data files, in exchange for efficiency in time and storage space, without using compression.

That's all.
Dynamic forms rendering based on preferences
Published: 2019-08-31 07:37:53

I am working on a product UI dynamic forms design.Where requirement is :
1. Develop master forms layout
2. Store it in DB in JSON format.
3. Admin use has access to this forms layout and can change the layout add or remove fields for a particular user
4. it is saved in DB
5. once User logs  UI is dynamically rendered based on what was saved by admin user above.

I would like to know how Angular and JSON can help in this.
Useful Docker shell aliases to speed up Development
Tags: docker, shell, bash, commandline
Published: 2019-08-31 07:37:30

I work with Docker all the time. Ditching my MAMP stack for Docker was one of the most convenient and useful decisions I've ever made. If you are interested in Docker you should take a look at my current 
[Docker Stack](https://blog.kovah.de/en/5gw1x8-a-drop-in-docker-stack-for-php-app/) which describes how I use Docker for
PHP projects.

## Useful Shell aliases

If you work with Docker you have to issue a lot of shell commands to start, stop and manage the containers. Most of the commands are usually targeted towards Docker Compose which is a managing layer for Docker. To improve working with the shell I created several aliases which shorten the amount of typing needed to perform the most common actions.

**Container management**

```bash
# Start the docker-compose stack in the current directory
alias dcu="docker-compose up -d"

# Start the docker-compose stack in the current directory and rebuild the images
alias dcub="docker-compose up -d --build"

# Stop, delete (down) or restart the docker-compose stack in the current directory
alias dcs="docker-compose stop"
alias dcd="docker-compose down"
alias dcr="docker-compose restart"

# Show the logs for the docker-compose stack in the current directory
# May be extended with the service name to get service-specific logs, like
# 'dcl php' to get the logs of the php container
alias dcl="docker-compose logs"

# Quickly run the docker exec command like this: 'dex container-name bash'
alias dex="docker exec -it"
```

**System-wide management**

```bash
# 'docker ps' displays the currently running containers
alias dps="docker ps"

# This command is a neat shell pipeline to stop all running containers no matter
# where you are and without knowing any container names
alias dsa="docker ps -q | awk '{print $1}' | xargs -o docker stop"
```

Those are the current aliases I use to speed up development. Feel free to extend them, please share them with me via Twitter.

---

_This article was first posted on [Blog.Kovah.de](https://blog.kovah.de/en/2019/useful-docker-shell-aliases/)_
Magento 2 Infinite Scroll
Tags: magento, webdev
Published: 2019-08-31 07:12:12

It is a key factor to provide a seamless customer experience which makes a fantastic e-commerce website. [Magento 2 Infinite Scroll](https://www.mageants.com/infinite-scroll-for-magento-2.html) helps to accomplish it for every e-commerce website.

**Key Features**

Loading pages automatically
Can be customized easily
Scroll Ajax without any interruptions
Increase in the conversion rates 
Text can be loaded easily

**Overview**

Magento 2 Infinite scrolling makes it easier and more pleasant to experience shopping. It can load products automatically without interruption. The client can see which page the item belongs to while scrolling down. They can readily recognize the position of the item.
Where do the PROs learn ?
Tags: discuss
Published: 2019-08-31 06:26:48

Short bio. My name Adizbek, I am 21, started programming ~5 years before.
Currently I work as fullstack developer(js) at local company.   

I like learning new technologies and all with related to good development techniques.

I have learnt many frameworks(php, js, python, java), SQL and NoSQL databases storages, but not enough dig deep, While I work I feel it's not enough, there should the better ways to write the code. 

My question is "Where do the PROs learn?" (I am not pointing myself as PRO 😅).  I feel that I want learn more and more, but can't find right place to go further, to dig deeper.

Please share your strategies or ways how to become better PRO below in comments. 👇


P.S. It's my first post on DEV community. 😅
What's the FASTEST web framework? | Web Framework Comparisons
Tags: react, vue, webdev, javascript
Published: 2019-08-31 05:40:40

---
title: What's the FASTEST web framework? | Web Framework Comparisons
published: true
description: Can we measure the fastest framework? Why yes we can, and I'm showing you how in today's video.
tags: react, vue, webdev, javascript
---

##Ever wonder what the fastest web framework is? 

In today’s video I’m showing you a GitHub repo that lets you challenge two or more backend frameworks on your command line. 

This can be a helpful tool when you’re trying to decide what backend language/framework may be the best for the job. 

But don’t worry, I’m also showing you the fastest frontend frameworks as well! 

GitHub repo: https://github.com/the-benchmarker/web-frameworks

Frontend Benchmarks: https://www.freecodecamp.org/news/a-realworld-comparison-of-front-end-frameworks-with-benchmarks-2019-update-4be0d3c78075/

***

_This video originally appeared on www.youtube.com/realtoughcandy_.
 Come follow me there!
CODELF DAU is staying in 2K for about 5 months
Tags: codelf, github
Published: 2019-08-31 03:04:32

Wow, CODELF DAU is staying in 2K for about 5 months, and the GitHub star is heading to 7.5K. I still don't know what to do with it. Looks like devs love it.  https://unbug.github.io/codelf/
Learn Coding Online
Tags: computerscience, beginners, career
Published: 2019-08-31 02:53:22

How to become a software developer?

Here is the list of websites where 
you can :
1 Learn concepts  
2 Practice coding  
3 Develop Applications

If you are beginners, here is the recommended set of websites:
1 W3 Schools Click Here: https://www.w3schools.com/
2 Geeks for Geeks Click Here: https://www.geeksforgeeks.org/
3 Tutorials Point Click Here: https://www.tutorialspoint.com/index.htm
4 Free Code Camp Click Here: https://www.freecodecamp.org/

If you are already a developer in the industry, here is the recommended set of websites:

1 Leet Code Click Here: https://leetcode.com/
2 Hacker Rank Click Here: https://www.hackerrank.com/
3 CodeAcademy Click Here: https://www.codecademy.com/learn 

Code Academy is the only website from list wants you to buy a membership to avail their entire features and the rest of them were free and easy to learn!
  
Do you have any tips or resources you would like to share?

Thank you
Happy Coding!
PowerShell Core | Out-GridView 😎Welcome Back😎
Tags: powershell, productivity, ptbr
Published: 2019-08-31 02:37:27

# Out-GridView no PowerShell Core

E ae, pessoal, beleza ? Vamos falar sobre uma feature muito bacana para quem utiliza o **PowerShell 5.1**. Esse cmdlet está disponível por padrão, porém, se você utiliza o PowerShell Core 6.X provavelmente notou que ele não estava disponível, mas isso acabou 🥳🥳🥳.

O grid é bem útil na visualização do output de objetos, pois conseguimos aplicar filtros,e é uma ótima opção para usuários com pouca intimidade com scripts. 

## Vamos então aos passos necessários para utilizar o Out-GridView.
1. Ter o PowerShell Core 6.2+ no seu SO. (disponível para Linux, Windows, macOS, everywhere)
2. E o módulo Microsoft.PowerShell.GraphicalTools.

{% gist https://gist.github.com/EwertonJordao/8ba7faaef23036e1d22c93dbcd68dc30.js %}

>Obs: No script presumo que você já tenha o **PowerShell Core** no seu SO, e que a PSGallery esteja como **provider trusted**, ao instalar um módulo ele vai buscar em um repositório que neste caso é a PSGallery.

```
Get-PSRepository
```
Output esperado após executar o cmdlet acima.
 
InstallationPolicy | SourceLocation                           | Name     
------------------ | ---------------------------------------- | ---------
**Trusted**           | https://www.powershellgallery.com/api/v2 | PSGallery


Caso o Output do cmdlet seja conforme tabela abaixo: 

InstallationPolicy | SourceLocation                           | Name     
------------------ | ---------------------------------------- | ---------
**Untrusted**          | https://www.powershellgallery.com/api/v2 | PSGallery


Definimos o valor para **Trusted** desta maneira:
```
Set-PSRepository -Name PSGallery -InstallationPolicy Trusted  
```
Exemplo Windows 10:
![Out-GridView em funcionamento no Windows 10](https://thepracticaldev.s3.amazonaws.com/i/vy36j9kwu18usfbuoz31.jpg)

Exemplo centOS:
![Out-GridView em funcionamento no centOS](https://thepracticaldev.s3.amazonaws.com/i/w8lh7i4lsde1kop1u2sk.png)

Fico devendo o output no macOS :/, desculpe.

É isso pessoal, não se esqueça, **PS é vida**.

>Curtiu? Quer saber mais? {% github https://github.com/powershell/GraphicalTools %}