teleren/devto huggingface dataset:
Pyspark Code Snippets
Tags: python, codenewbie, 100daysofcode
Published: 2020-05-28 08:55:25

I have been using PySpark for some time now and I thought to share with you the process of how I begin learning Spark, my experiences, problems I encountered, and how I solved them! You are more than welcome to suggest and/or request code snippets in the comments section below or at my twitter [@siaterliskonsta](https://twitter.com/siaterliskonsta). I will keep this post maintained and update it once I create more gists.

The original post is located at [my blog](https://thelastdev.com/pyspark-code-snippets/).

DataFrames
==========

From Spark's website, a DataFrame is a *Dataset* organized into named columns. It is conceptually equivalent to a table in a relational database or a data frame in R/Python, but with richer optimizations under the hood. DataFrames can be constructed from a wide array of [sources](https://spark.apache.org/docs/latest/sql-data-sources.html) such as structured data files, tables in Hive, external databases, or existing RDDs.

Reading Files
=============

Here are some options on how to read files with PySpark.
{% gist https://gist.github.com/siakon89/ff0f3c79f000cdb304e2bdf185b977fb %}

Selecting Columns
=================

Now that you have some data in your DataFrame you may need to select some specific columns instead of the whole thing. This is how you do it
{% gist https://gist.github.com/siakon89/ecfbb7cb3ff7afb3f10dc2ffec29fd15 %}

Filtering
=========

In addition, you may also want to filter the rows based on some conditions. This is how it works.
{% gist https://gist.github.com/siakon89/c470fb04187fc672d425d593b3e2b420 %}

GroupBy
=======

Now that you have learned the basic column and row manipulation, let's move on to some aggregations.

{% gist https://gist.github.com/siakon89/5c4ff88fdd14595b76524da0f6ee0b25 %}

RDDs - Resilient Distributed Datasets
=====================================
RDDs are fault-tolerant collection of elements that can be operated on in parallel. More information can be found in the official spark documentation [here](https://spark.apache.org/docs/latest/rdd-programming-guide.html#resilient-distributed-datasets-rdds)

Reading Files
=============

Let’s see how to read files into Spark RDDs (Resilient Distributed Datasets)

{% gist https://gist.github.com/siakon89/01e61eb10db7309e8fca30d5a76b735b %}
My First ES6 Test Code - Trying Modules without compilers (Export, Import)
Tags: es6, javascript, modules
Published: 2020-05-28 08:39:19

With the basic knowledge of Javascript & jQuery, I decided to step in to ES6. What excites me is the Module where I can import & export multiple JS files. Currently its hard to manage when there are lot of functions in one JS file which I used to do using jQuery. 

Since all modern browser now supports ES6, I decided not to compile and see how it runs in the browser.

**I started with a basic HTML**

```html
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Document</title>
</head>
<body>
    <h1>Hi</h1>
    <script src="./main.js" type="module"></script>
</body>
</html>
```

Then I created 3 modules and one `main.js` to include in my HTML Page. 

**a.js**
```javascript
import { stuff } from "./b.js";
import { randhex, validURL } from "./c.js";

stuff("It works");
 
console.log(import.meta); 
console.log(validURL("google.com/")); 
const goolink = validURL("google.com/");

console.log(randhex(5)); 

export { validURL, goolink };

```
**b.js**
```javascript
const stuff = function (text) {
  console.log(text);
};

export { stuff };

```
**c.js**
```javascript
const validURL = function (str) {
  var pattern = new RegExp(
    "^(https?:\\/\\/)?" + // protocol
    "((([a-z\\d]([a-z\\d-]*[a-z\\d])*)\\.)+[a-z]{2,}|" + // domain name
    "((\\d{1,3}\\.){3}\\d{1,3}))" + // OR ip (v4) address
    "(\\:\\d+)?(\\/[-a-z\\d%_.~+]*)*" + // port and path
    "(\\?[;&a-z\\d%_.~+=-]*)?" + // query string
      "(\\#[-a-z\\d_]*)?$",
    "i"
  ); // fragment locator
  return !!pattern.test(str);
};

const randhex = function (len) {
  var maxlen = 8,
    min = Math.pow(16, Math.min(len, maxlen) - 1),
    max = Math.pow(16, Math.min(len, maxlen)) - 1,
    n = Math.floor(Math.random() * (max - min + 1)) + min,
    r = n.toString(16);
  while (r.length < len) {
    r = r + randHex(len - maxlen);
  }
  return r;
};
export { randhex, validURL };

```
**main.js**
```javascript
import { goolink } from "./a.js";

if (goolink) {
  console.log("yaay");
}

```

## Result

Here's what I got in `console.log` It works and runs in the browser without any issues. No compilers, No Babel, No Webpack nothing. Just pure Javascript. 

```console
It works
> Object
true
yaay
```

## What I learned

1. ES6 Modules will only work on a Server like `localhost` or `web server`. It won't work via `file:///`
2. While adding `<script>` tag to HTML, we must add `type="module"` attribute. Otherwise the module won't work. 

## Your Suggestions
Do you have any suggestions on how to improve my code? or any other stuff I should be aware of? Let me know in the comments.
I'm Giving My First-Ever Conference Talk!
Tags: devjournal, codenewbie, womenintech, codeland
Published: 2020-05-28 08:20:56

_This post was originally published on May 28, 2020 on [my blog] (https://alexlsalt.github.io/blog)._

Alright, self - enough lollygagging around. It's been just about an hour since I sat myself down to write and in that hour, I've successfully perused a few quick YouTube videos, subscribed for a royalty-free music site, and checked Twitter a handful of times.

So! Big news since the cat's now outta the bag: I'm going to be giving my first-ever talk at the CodeLand Conference in July!

![Screenshot from Twitter](https://dev-to-uploads.s3.amazonaws.com/i/qfevain0mgrafw2flxkm.png)

I'm seriously so excited! I posted back at the beginning of April about submitting a proposal to this conference - and I feel so great to have gotten the spot. I can't wait.

Here's the Alex from April 1:

> _Today's been a much slower day, and it's really felt good! Last night, I'm proud to have sent off my finished application to give a talk at this year's virtual CodeLand Conference! I've never submitted a proposal for a conference CFP, and I can gladly say that I've crossed that one off my coding-journey bucketlist._

> _Regardless of the outcome, I'm happy to have put my foot forward, and I'm confident I'll give a talk about coding at some point in the next few months, whatever that may look like!_


My talk, titled *Being Utterly Fearless in Your Pursuit of Learning to Code*, will mostly be about my journey and my five greatest tips for others who've ventured into the unknown to pursue learning to code and transitioning their careers.

Can't wait to share! Want to attend the conference? [Head over here >>] (https://codelandconf.com/)

[Now let's be friends over on Twitter >>] (https://twitter.com/alexlsaltt)
Reconfiguring your journey to Data Science
Tags: datascience, machinelearning
Published: 2020-05-28 07:52:04

Nothing is better than writing what you have learnt, the journey was quite tough, but yes, `YOU can do this`.
I have written a blog on __"Reconfiguring your journey to Data Science"__ . The blog will guide you when to jump on this technology and when to prepare more before starting with Data Science. 

Do read this and let me know your feedback.
{% medium https://medium.com/bellatrixdata/reconfigure-your-journey-to-data-science-44092971082c %}

I will be quite happy if my work helps you in any way. Let me know about this too.

If you want me to write on any special segment, comment below, I will look into it.
Quad Tree Basics
Published: 2020-05-11 23:33:15

Trees, oh glorious trees, they provide us shade, oxygen, and a way to optimize our collision system. Recently at Wooden H Studio we have been working on trying to implement a Quad Tree system to help optimize our mesh collisions.

## What’s a Quad Tree?

A Quad Tree is a tree that uses a recursive pattern that subdivides itself into 4 other quad trees, insert a point into itself or one of its children, and query how many point are within a defined region like a rectangle or a circle, this type is mainly used for 2D applications. Additionally, quad trees have a long lost brother named Octrees, the main difference between them is rather than splitting into 4 sections it splits into 8 sections and is mainly used for 3D applications.
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/ub0hsgpipg2cy5n12rc9.jpg)

##So how do you use one?

The way both of these trees work is they have a capacity for how many objects can be inside a section at each time for example we will assume our capacity is 3. As long as we stay underneath or equal to the capacity we do not need to divide our tree
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/ytp4n6dp3dz19vpbg4bd.jpg)
But, once we go over that capacity we need to subdivide. Regardless of where the new point is it is prefered that we split it evenly to avoid confusion.
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/37n5u1aoxemfeh3bz90h.jpg)

##How does this help us?

Well as I said before this will help us optimize our newest problem, mesh collisions. Now we could use an Octree for this but the problem is we have very limited memory and while this is pretty fast it does use a fair amount of memory. The other reason why we opted for the quad tree is the only thing we are using the mesh collision for is the ground. Since our ground is going to be hilly and rugged we find the centroid point of our triangle and insert it into our quad tree. Since we don’t plan on our environment moving around and changing we can pre-compute this before the game begins and call a query to reduce the amount of collision checks. With this knowledge we can reduce collision checks from 200 to possibly 5, with very little complexity or even effort.

By Gage Dietz
Boost your productivity with this tiny IntelliJ plugin! 
Tags: productivity, intellij, java, javascript
Published: 2020-05-28 07:09:01

As a developer i think our job is not just writing code but looking for the best way to do so. Writing good code usually takes a lot of searching on the internet. So every time i have a doubt on something, i have to select text i want to look for, open the browser and dig the internet until i find something useful. 

So i thought, why i don't implement this directly in my IDE?

And that's how [DevSearch for IntelliJ](https://plugins.jetbrains.com/plugin/14298-devsearch) was born! 

DevSearch it's a small plugin to search the right thing from the right context. If you select some text in the console, it will propose to search for it on StackOverflow. If you select a folder under node_modules, it will suggest to search for it on NPM. And these are just a couple of examples. 

At the moment the plugin supports few integrations, but it's [open source](https://github.com/corradodellorusso/devsearch) and lets you write your own integration with less than 10 lines of code! 

Any contribution would be very appreciated.

Here are the links:
Plugin: https://plugins.jetbrains.com/plugin/14298-devsearch
GitHub: https://github.com/corradodellorusso/devsearch
How to Put Your TYPO3 Website in Maintenance/Offline Mode
Published: 2020-05-28 07:05:20

If you want to perform a maintenance task, you may need to put your TYPO3 website in maintenance mode. This state means your website is unavailable for some time.
What is TYPO3 maintenance mode and how to put your TYPO3 website in maintenance mode? We will help you understand everything you need to know about the TYPO3 maintenance mode in this blog. We will also share ways to make your TYPO3 website under maintenance mode more appealing.

- What is the TYPO3 maintenance mode?
- When should I set my TYPO3 website to maintenance mode?
- How to Put TYPO3 website in Maintenance Mode? Three Easy-Methods!
- Method 1 - Using TYPO3 Core-Way
- Method 2 - Using Maintenance Mode TYPO3 Extension
- Method 3 - Using Maintainance Mode TYPO3 Template

Read the complete blog at https://t3terminal.com/blog/typo3-maintenance-mode for an in-depth tutorial.
Work From Home Vs Office: Which One To Choose Post Lockdown?

Tags: workfromhome, remotework, development, developers
Published: 2020-05-28 07:01:04

Because of the Covid-19 pandemic situation, many industries opted for the trending work culture which is Work From Home and this comes along with the rapid development in technology. This shows another face of work culture which is to communicate with their teams and supervisors regardless of their geographical areas. You can easily communicate with your teams because of B2B software and your work tools that are accessible regardless of your location. This gives you the flexibility to adjust your workflow as per your priority, wherever you are. This results in more job opportunities that may come your way. Working from home might come with a mixed blessing. Hence here we are comparing the office work and remote work with some benefits and obstacles of each so as to help you to decide Work From Home vs Office post lockdown.

**Work From Home vs Office-**
*1. Dress code-*
Work from Office-

When you work from the office, it is expected that you must dress up according to your profession, business industry, or location. Generally, you must be dressed up as per the standards defined by an organization which is usually formal clothes. One may get irritated by the formal clothes and results in its productivity throughout the day. According to a survey, it has been analyzed that a company that has decided the dress code is perceived negatively by 61% of employees looking for a job change.

Work from Home-

When it comes to remote working, you have 100% freedom to wear the clothes that you’re comfortable in. And it has been noted that employees are more productive when the dress code is relaxed.

**2. Flexibility Of Work Hours-**
Work from Home–

We have mentioned in the introduction that remote working provides you the flexibility to schedule the work according to your priority. Obviously, you must be in touch with your team and accordingly schedule the team meetings, in queue tasks and end of the reporting but still, there is much more freedom than an office day.

Working remotely allows you to schedule your work hours as per your productivity peak hours whether it is early morning or late at night. And hence you can manage the work more productively when you are energetic which results in effective work done.

Work from Office-

Working from an office requires you to align your work between the specific office timings. And it is expected that you must be prompt at that time. But the truth is, you can not be effective all the time. This means you have to deal with your habits and nature with no choice of freedom to set the schedule.

**3. Governing By Someone-**

Know more at- [https://solaceinfotech.com/blog/work-from-home-vs-office-which-one-to-choose-post-lockdown/]
Merge ConfigMap and Secrets
Tags: kubernetes, secret, config, ruby
Published: 2020-07-06 09:06:10

---
title: Merge ConfigMap and Secrets
published: true
date: 2020-05-28 07:00:00 UTC
tags: #kubernetes #secret #config #ruby
canonical_url: https://storage-chaos.io/configmap-and-secret.html
---

# TL; DR

To use a [Secret](https://kubernetes.io/docs/concepts/configuration/secret/) value within a [ConfigMap](https://kubernetes.io/docs/concepts/configuration/configmap/) you can use an [initContainer](https://kubernetes.io/docs/concepts/workloads/pods/init-containers/) to call a [template engine](https://en.wikipedia.org/wiki/ERuby).

# The premise

In the [previous post](kubernetes-event-monitoring.html), I presented how to use kubernetes-event-exporter with Elasticsearch.

One of the problems I faced is that the tool doesn’t follow the configuration guidelines from the [12-factor app](https://12factor.net/config) methodology.

That is to say, we have to put the credentials in the YAML configuration rather than in environment variable :-(

As for Kubernetes, it doesn’t allow us to mix Secret values within ConfigMap.

# The solution

To solve that issue, we have 3 components:

1. the `Secret` as-is
2. the `ConfigMap` which will have the configuration as a **template**
3. the `initContainer` that will merge the two

## SecretMap

The secret comes from the [ECK Operator](https://www.elastic.co/elastic-cloud-kubernetes) ; we can get it with `kubectl get secrets quickstart-es-elastic-user -o yaml` :

```
apiVersion: v1
kind: Secret
data:
  elastic: YU80bnc4NzZWMXBWMThOZThqOFlnOE1r

```

## ConfigMap

In the case of [k8s-events-reporting](https://github.com/coulof/k8s-events-reporting) the ConfigMap looks like this:

<noscript><pre>400: Invalid request</pre></noscript><script src="https://gist.github.com/a9c73d015facf180a70d3e9dee4bd101.js?file=configmap.yaml"> </script>

The important piece is the last line, `<=% %>` is the erb syntax to call ruby code, and [ENV](https://ruby-doc.org/core-2.7.0/ENV.html) is a hash to acces the environment variables.

## Why [ERB](https://docs.ruby-lang.org/en/master/ERB.html)?

1. Because I ♥ Ruby !
2. Because the erb command line comes with the [ruby docker official image](https://hub.docker.com/_/ruby) (there is no need for a custom Dockerfile and therefore no maintenance)

## initContainer

Last but not least, here is the `Deployment` with the initContainer `config` that will craft the config file from both the `Secret` passed as an environment variable and the `ConfigMap` template. The `event-exporter` container can later use that file.

<noscript><pre>400: Invalid request</pre></noscript><script src="https://gist.github.com/a9c73d015facf180a70d3e9dee4bd101.js?file=deploy.yaml"> </script>
Difference between Controllers, Routes and Services
Tags: help, express, node, discuss
Published: 2020-05-28 06:52:27

I am building an API service using Nodejs, Expressjs and MongoDB. Also, I am using Mongoose library to deal with database. After going through several blogs on file structuring, I am quiet confused about it. I want help with the file structuring for my project. My "src" folder structure looks like this so far :- <br />

```
src
  |
  |__Controllers
  |      |
  |      |__ admin.js
  |      |__ customer.js
  |__Models
  |__Public
  |__Routes
  |      |
  |      |__ admin.js
  |      |__ customer.js
  |__Views
  |__Services
         |
         |__admin.services.js
         |__customer.services.js
```
My concerns are :-
1) Are Controllers the same as Routes? If not, should they be put in Routes folder?<br />
2) How should my Service files be structured (Am I doing this right?)? Or should each API call get a different service file? <br />
3) Since routes are handling request and response objects, what do we use controllers for? Can we just call "services" from routes and respond from each API route, or am I leaving something behind?<br />
Python Tricks - Functions to change Data Type
Tags: python, programming, software
Published: 2020-05-28 06:51:46

1. int () to convert to integer;
2. long () to change to an integer length;
3. float () to change to float;
4. bool () to change to boolean;
5. chr () to change to character;
6. str () to convert to a string.
7. bin () to convert to Binary numbers.
8. hex () to convert to Hexadecimal numbers.
9. oct () to convert to octane numbers.

Thanks for read my docs, hope enjoy it.
Internshala Training Program 
Tags: internshala, training, internship, isp
Published: 2020-05-28 06:42:53

Hey guys!!
Would you like to learn a new skill before the college reopens? Because Internshala Trainings is offering 20+ online trainings which can be learnt from the comfort of your home. So, invest in yourself, learn an in-demand skill and earn a prestigious certificate. Also, avail a limited period *55%+10%* discount through my link here https://internshala.com/i/TFC-ISP18KISH7415
Vanilla JavaScript Chunk Array
Tags: javascript
Published: 2020-05-28 06:36:30

The other day I needed to chunk an array of over a million records in batches of a maximum of 5000.

There are multiple correct ways; I'm going to show you the one I used for chunking an array.

## Chunk Array in JavaScript

```js
var input = new Array(10001);

function chunkArray(array, chunk_size) {
  var results = [];

  while (array.length) {
    results.push(array.splice(0, chunk_size));
  }

  return results;
}

var result = chunkArray(input, 5000);
console.log(result.length);
// 3
```

So we start by defining a new array of `10001` records.
Then we define our `chunkArray` function; it will accept an input array and a chunk_size parameter.
Inside the `chunkArray` function, we create a new results array and `while loop` over the input.
We then push the amount of `chunk_size` into our result.

Then we call the function passing it our two parameters. In this case, the result will have three arrays, two of 5000 records, and one with one record.

You can view this in action on Codepen.

<p class="codepen" data-height="265" data-theme-id="dark" data-default-tab="js,result" data-user="rebelchris" data-slug-hash="PoPvePB" style="height: 265px; box-sizing: border-box; display: flex; align-items: center; justify-content: center; border: 2px solid; margin: 1em 0; padding: 1em;" data-pen-title="Vanilla JavaScript Chunk Array">
  <span>See the Pen <a href="https://codepen.io/rebelchris/pen/PoPvePB">
  Vanilla JavaScript Chunk Array</a> by Chris Bongers (<a href="https://codepen.io/rebelchris">@rebelchris</a>)
  on <a href="https://codepen.io">CodePen</a>.</span>
</p>
<script async src="https://static.codepen.io/assets/embed/ei.js"></script>

### Thank you for reading, and let's connect!

Thank you for reading my blog. Feel free to subscribe to my email newsletter and connect on [Facebook](https://www.facebook.com/DailyDevTipsBlog) or [Twitter](https://twitter.com/DailyDevTips1)
Venturing into front-end land
Tags: diaristic
Published: 2020-05-28 06:05:26

I've been a professional software developer for almost three years now, and I'm still super insecure about my front-end skills. So insecure to the point where at work I make a conscious effort to avoid front-end related tasks by citing my specialty and affinity for the back-end which is a complete lie; I like the back-end because most of my professional training and experience has been back-end oriented.

Recently, I've summoned up the courage to get out of my comfort zone and take a few courses on JS, CSS and HTML, and they've certainly helped me feel at ease about rendering something on the browser. Strangely it took me this long to discover there was a huge merit to working in the front-end: getting to interact directly with what I'm building. In front-end land, I know what I'm building is pretty ugly (😅), but at least I'm directly responsible for the mess, whereas it can be a muddy process tracking exactly what it is that I do on the back-end.

So where am I at? Well, I think I have a loose grasp of everything in JS land. A mentor advised me to start blogging to reinforce the concepts, so here I am. Over the past three weeks, I've learned tons of new things, and I thought it would be nice to share them here. You can tell me if what I learned is BS. For reference, here are some of the topics I intend to cover herein:

- JavaScript
- React
- Vue
- D3

As for how my newfound skills will help me professionally, I'm not sure. Until I get enough practice and feel like I really know what I'm doing, nothing's going to change at work; I'll continue trodding along in Clojure land which has been my favorite land as of recent. Anyways, I hope you enjoy my writing.

Warmly,
EK