teleren/devto huggingface dataset:
Is Machine Learning the goal of Data Science?
Tags: python, machinelearning, ai, aws
Published: 2020-04-08 06:33:02

Welcome to another episode of our ‘Machine Learning’ Series.

So is Machine learning the ultimate goal of any data science initiative? It seems to be so the in the recent past where a quaint little area of Data Analysis is thrown into the cauldron of Machine Learning and AI madness. Folks (I admit including myself) got into to this area only due to the allure of Machine Learning and AI. It is like the bait that is catching and reeling us all in. I was listening to this podcast ‘path to a data science career’.  In this podcast, they were talking about what does one need to know to build a career in data science. The most important take away for me from this pod cast is their discussion on if you need Machine Learning skills to be a good data scientist. The answer is NO. Or it is a qualified ‘it depends’.

Data Science is a vast inter disciplinary area. Some folks come to data science from software programming background like me. Some can do the reverse and come from math or statistical background. Or others from a technical field like biology or physics. So a team of Data Analysts will usually have a back ground from diverse background that complements each other. So you can be wherever you want to be in the Data Science pipeline shown in the figure below.  You could be a great at Pandas package in Python and that may be all you do. If you can do it really well and build some great analysis packages for your colleagues to use. You could be a great report writer with skills in data visualization and storytelling. You can bring some technical background in understanding the data domain, say a genetists or bio informatician. If you are machine-learning engineer you can be building some interesting models on the data for better understanding, predicting, classifying or forecasting. Still you will only be part of this bigger team of experts and will depend on and complement their expertise.

No doubt with the newer ML algorithm and awesome processing power at our disposal, our ability to analyze and understand huge amounts of data (Big Data, another catchy bait reeling some of us in). Still most of the work of the data scientist/analyst is still being done in collecting, aggregating, filtering, transforming, hypothesizing and report writing aspects of Data Science. All these steps are critically important even if Machine Learning is the team’s ultimate goal. So pick a spot in this pipeline that interests you and where you can develop a deep understanding and experience. Don’t be a shallow grazer and a Machine Learning wanna be just because that is what the latest buzz is all about.
How to maintain a huge repo with sporadic access
Tags: help
Published: 2020-04-08 06:15:54

In our team we have the healthy habit of automating our process in simple ways.

One simple way of doing it is to create **scripts**, snippets of code that are just run from a developer computer. It could include things like calling an API and writing a CSV file from there. We called them scripts but it can be called "CLI apps" if you prefer it. Some scripts we use them only once, others we use them every now and then (like once in a month)...

We have one repo with several of those scripts. Each script is in a directory. All are in Node.js so every directory has a `package.json` file. To run it, you simply do `cd «directory»; npm install; node .`

How do you maintain and keep updated a repo like that? We don't want to check every dependency of every directory... Maybe a monorepo?

All suggestions are welcome :)
Not able to do 301 redirection if slug contains '+' eg: https://www.hellno.com/foo+foo to https://www.hellno.com/foo-foo
Tags: help
Published: 2020-04-08 06:13:21

Not able to do 301 redirection if slug contains '+'.
eg: https://www.hellno.com/foo+foo to https://www.hellno.com/foo-foo
Using Broadway and RabbitMQ to Create a Data Pipeline in Elixir
Tags: softwaredevelopment, programming, rabbitmq, elixir
Published: 2021-08-06 03:31:32

---
title: Using Broadway and RabbitMQ to Create a Data Pipeline in Elixir
published: true
date: 2020-04-08 05:50:45 UTC
tags: softwaredevelopment,programming,rabbitmq,elixir
canonical_url: 
---

[![](https://cdn-images-1.medium.com/max/1293/0*HwWtfo9OudnUZhUR.png)](https://itnext.io/using-broadway-and-rabbitmq-to-create-a-data-pipeline-in-elixir-900ddff57c7d?source=rss-9a53804556------2)

In this article, I will focus on creating a data ingestion and processing pipeline for Elixir using Broadway [1] library. As per…

[Continue reading on ITNEXT »](https://itnext.io/using-broadway-and-rabbitmq-to-create-a-data-pipeline-in-elixir-900ddff57c7d?source=rss-9a53804556------2)
Weekly web development update #20
Tags: weekly, webdev
Published: 2020-04-08 05:36:02

Hello everyone - here's the new weekly update of things around webdev.

______

##[Open UI](https://open-ui.org/?utm_source=wweb.dev)
[![Open UI](https://wweb.dev/weekly/screenshots/openui.png)](https://open-ui.org/?utm_source=wweb.dev)
An open-source project to maintain a standard for UI and to promote its adherence and adoption.

______

##[AllTheFreeStock](https://allthefreestock.com/?utm_source=wweb.dev)
[![AllTheFreeStock](https://allthefreestock.com/assets/images/twitter-banner.png)](https://allthefreestock.com/?utm_source=wweb.dev)
A large list of free stock images, videos, music and more.

______

##[HTML DOM](https://htmldom.dev/?utm_source=wweb.dev)
[![HTML DOM](https://htmldom.dev/assets/screenshot.png)](https://htmldom.dev/?utm_source=wweb.dev)
A big list of vanilla JS scripts for the most common DOM interactions.

______

##[Magic](https://magic.link/?utm_source=wweb.dev)
[![Magic](https://wweb.dev/weekly/screenshots/magiclink.png)](https://magic.link/?utm_source=wweb.dev)
A cool project to add passwordless logins to your website.

______

##[Flowify](https://etesam.surge.sh/#/flowify?utm_source=wweb.dev)
[![Flowify](https://wweb.dev/weekly/screenshots/flowify.png)](https://etesam.surge.sh/#/flowify?utm_source=wweb.dev)
A browser extionsion for creating free flowing startpages.

______

##[Vuln Cost](https://snyk.io/security-scanner-vuln-cost/?utm_source=wweb.dev)
[![Vuln Cost](https://snyk.io/wp-content/uploads/vuln-cost-animation.gif)](https://snyk.io/security-scanner-vuln-cost/?utm_source=wweb.dev)
A cool VS Code extension to show in-line how many vulnerabilities a package contains.

______

##[Pluralsight #FreeApril](https://www.pluralsight.com/?utm_source=wweb.dev)
[![Pluralsight #FreeApril](https://www.pluralsight.com/content/pluralsight/en/jcr:content/image-res/file.transform/share-image/image.img.552aecd7-9b29-438a-9329-ab30ac2e772e.jpg)](https://www.pluralsight.com/?utm_source=wweb.dev)
Pluralsight is offering all of their video courses for free this april.

______

##[Lisan](https://lisanjs.com/?utm_source=wweb.dev)
[![Lisan](https://lisanjs.com/img/lisan-og.png)](https://lisanjs.com/?utm_source=wweb.dev)
A fast i18n library for adding multiple languages to your application.

______

##[How to SSH Properly](https://gravitational.com/blog/how-to-ssh-properly/?utm_source=wweb.dev)
[![How to SSH Properly](https://gravitational.com/blog/images/2020/how-to-ssh-properly-header.png)](https://gravitational.com/blog/how-to-ssh-properly/?utm_source=wweb.dev)
A good guide about SSH by Gus Luxton on Gravitational.

______

To see all the weeklies check: [wweb.dev/weekly](https://wweb.dev/weekly)
How to use Persistent Storage in OpenShift?
Tags: devops, openshift, kubernetes, mongodb
Published: 2020-04-08 04:48:20

## Objective
- Deploy **Mongodb** Application
- Create **PersistentVolumeClaim**
- Assign **PersistentVolumeClaim** to **Mongodb** Application
- Verify that data gets stored in Persistent Storage

### Step 1: Set up OpenShift environment
Go to [Katacoda.com](https://katacoda.com/openshift/courses/playgrounds/openshift39) & click on start scenario

### Step 2: Update environment permissions
```
setenforce 0
```

### Step 3: Deploy mongodb application
```
oc new-app --name mongo -l app=db --docker-image=centos/mongodb-36-centos7 -e MONGODB_ADMIN_PASSWORD=secret
```

### Step 4: List pods
```
oc get pods
```

### Step 5: Describe pod
```
oc describe pod <POD_NAME>
```

### Step 6: Create PersistentVolumeClaim & assign it to mongodb application
```
oc set volume dc/mongo --add --name=<PVC_NAME> -t pvc --claim-size=10Gi  --overwrite --claim-mode="ReadWriteMany"
```
```
Example: oc set volume dc/mongo --add --name=mongo-volume-1 -t pvc --claim-size=10Gi  --overwrite --claim-mode="ReadWriteMany"
```

### Step 7: List PersistentVolumeClaims (PVC's)
```
oc get pvc
```

### Step 8: List PersistentVolumes (PV's)
```
oc get pv
```

### Step 9: Describe pod
```
oc describe pod <POD_NAME>
```

### Step 10: Describe PersistentVolume (PV)
```
oc describe pv <PV_NAME>
```

### Step 11: List files
Go to PV location
```
ls
```

{% youtube wJ1jCpj1Hbc %}
Node Fundamentals: Buffer
Tags: node, javascript, buffer
Published: 2020-04-08 04:44:21

This is a series of posts that will illustrate the what, why and how of Node. I'll be sharing my learnings from a course on Advanced NodeJS by Samer Buna offered on [PluralSight](https://www.pluralsight.com). Any code samples tagged or attached will be available at the following repo. {% github jscomplete/advanced-nodejs no-readme %}

# Buffer

Buffer is heavily used in Node to work with binary streams of data. It is a low-level object to represent a sequence of binary data.

A buffer is essentially a chunk of memory allocated *outside v8 heap* and we can put some data in memory, which can be interpreted in many ways based on the length of each character. That's why there is always a corresponding *character-encoding* associated with that buffer. 

Whatever we place inside a buffer, doesn't have any character encoding, so to read it we need to specify an encoding.

Unlike arrays, once the buffer is allocated, it can't be resized. We can create a buffer in 1 of 3 major ways.

1. Buffer.alloc(n) - Allocates a 0-filed buffer of n bytes in memory.
2. Buffer.allocUnsafe(n) - Allocates a buffer of n byte in memory. This can cause a vulnerability as it can contain sensitive information.
3. Buffer.from() - Allocates a buffer with a value passed in argument.

```js
const string = "touché";
const buffer = Buffer.from("touché");
console.log(string.length) // 6
console.log(buffer.length) // 7
```

Buffers are useful when we want to read an image file from a TCP stream or a compressed file or any other form of binary data.

Just like arrays and string, we can use operations like `includes`, `slice`, `indexOf`.

In the case of `slice`, unlike arrays, a sliced copy will use the same memory space.

{% replit @khpatel4991/ClumsyQuarrelsomeMacro %}

## String Decoder

When converting streams of binary data, use [String Decoder](https://nodejs.org/api/string_decoder.html) module as it handles multi-byte characters much better. It gracefully handles incomplete characters, while calling `toString` method on buffer doesn't do that.

{% replit @khpatel4991/AwareIncompatibleWamp %}
Secure Code Guideline
Tags: security, codequality, codenewbie
Published: 2020-04-08 04:42:48

[Secure Coding Guideline Tip]
Tip1:
Input Validation of ALL types of input. This includes from your database, URL parameters and your database. Valid data on the backend (web server), not in the front end (javascript).
#tip1
#securecode
#Guideline
#tipandtrick
Node Fundamentals: CLI and Repl
Tags: node, javascript, cli
Published: 2020-04-08 04:30:36

This is a series of posts that will illustrate the what, why and how of Node. I'll be sharing my learnings from a course on Advanced NodeJS by Samer Buna offered on [PluralSight](https://www.pluralsight.com). Any code samples tagged or attached will be available at the following repo. {% github jscomplete/advanced-nodejs no-readme %}

# Node CLI and REPL

Node CLI comes with a variety of options to expose built-in debugging, multiple ways to execute scripts and other helpful runtime options.

Running a `node` command without any arguments starts a REPL.

R - Read
E - Eval
P - Print
L = Loop

When in REPL, you hit enter, it reads the command, executes it, prints the result and waits for the next command.

### Helpful CLI Tips and Tricks

- `-c` - Syntax check
- `-p` - Print Command. e.g `node -p "process.argv.slice(1) test 42"` will print ['test', '42']

### Helpful Repl Tricks and Tips

- Autocomplete by `Tab`

- `rlwrap` utility to track reverse search.

```bash
NODE_NOREADLINE=1 rlwrap node
```

- `_` is used to capture the last evaluated value.

- Special commands that begin with a `dot`. 
    - `.help` to print all such commands. 
    - `.break` to break out of a multiline session.
    - `.load` to load external script file
    - `.save` to save current session

- You can create your own repl with custom options by requiring a `repl` module and starting it with [custom options](https://nodejs.org/api/repl.html#repl_repl_start_options). You can also control repl's global context in case of preloading a library of data.

Example below will start the repl in strict mode and doesn't print anything when result is `undefined`. Also, it will have lodash available globally.

  ```js
  const repl = require("repl");
  const lodash = require("lodash");

  let r = repl.start({
    ignoreUndefined: true,
    replMode: repl.REPL_STRICT_MODE
  });

  r.context.lodash = lodash;
  ```
NodeJS Express API Endpoint Caching
Tags: node, express, cache
Published: 2020-04-08 04:28:06

Consider an endpoint is being accessed as frequent as your heartbeat by thousands users or more. How much do you trust the health of the endpoint to serve every users with no issues? If you do have system observability set up, you might probably see some timeout once a while from the endpoint and that could happen to one of your paying users. Even with the robust backend infrastructure set up, you could reach 99.9% uptime, what about the 0.1% of your user traffic? 

Caching with Redis come to the rescue! Given that this endpoint must not be real-time sensitive in order for cache to make senses. Such as weather info endpoint, since weather won't fluctuate in the matter of seconds.

Example here will be an Express App:
We have this controller just to pipe through the request and handling error when necessary.

```javascript
app.post('/getStatus', async (req, res) => {
	try {
		const cityId = req.body.cityId;

		const response = await weatherAPI.cacheAwareGetStatus(cityId);
		return res.status(200).json(response);
	} catch (e) {
		console.log(`Error calling API`);
		res.sendStatus(error.status);
	}
});
```

Then the API integration with external weather api. Cache layer is performed here.

```javascript
class WeatherAPI {
  public async getWeather(cityId) {
	const body = {
		cityId,
	};
	return await axios({
		url: 'api.weather.com',
		method: 'post',
		data: body,
	});
  }

  public cacheAwareGetStatus() {
    let raceResult;
    const cacheFetchTimeout = 5000;
    const defaultResponseData: GetPaymentInfoResponse = {
	  temperature: 0,
	  humidity: null,
	  wind: null,
    };

    const timer = new Promise((resolve) => {
	setTimeout(resolve, cacheFetchTimeout, { timeout: true });
});

	raceResult = await Promise.race([weatherAPI.getWeather(cityId), timer]);

	if (raceResult.timeout) {
		try {
			const weatherInfoCache = await redisClient.get(cityId);

			if (weatherInfoCache) {
				return JSON.parse(weatherInfoCache);
			} else {
				console.log(`API Timeout at ${cacheFetchTimeout} and cache not found`);
				return defaultResponseData;
			}

		} catch (e) {
			console.log(`API Timeout at ${cacheFetchTimeout} and error retreiving cache response`);
			return defaultResponseData;
		}
	} else {
		try {
			await redisClient.set(cityId, JSON.stringify(raceResult));
		} catch (e) {
			console.log(`Failed to set cache proceed to return response`);
		}
		return raceResult;
	}
  }
}

export const weatherAPI = new WeatherAPI();
```

`Promise.race([weatherAPI.getWeather(cityId), timer])` in the code determine whether the timer is up, probably the request is lost or still in flight for too long. We then want to get the cached response from Redis and return to the client. Otherwise, we proceed with the return of the latest response and update the cache in Redis. Sweet!
Node Fundamentals: Process
Tags: node, javascript, process
Published: 2020-04-08 04:26:36

This is a series of posts that will illustrate the what, why and how of Node. I'll be sharing my learnings from a course on Advanced NodeJS by Samer Buna offered on [PluralSight](https://www.pluralsight.com). Any code samples tagged or attached will be available at the following repo. {% github jscomplete/advanced-nodejs no-readme %}

# `process` Object

`process` object provides a bridge between a Node application and its running environment.

`process` object is an instance of event-emitter. So we can emit and listen to events on the object

## Useful properties and events

- pid: Outputs the process id on OS level.

- version: Read dependencies and versions

```bash
$ node -p "process.version"
{
  node: '13.12.0',
  v8: '7.9.317.25-node.30',
  uv: '1.35.0',
  zlib: '1.2.11',
  brotli: '1.0.7',
  ares: '1.16.0',
  modules: '79',
  nghttp2: '1.40.0',
  napi: '6',
  llhttp: '2.0.4',
  openssl: '1.1.1e',
  cldr: '36.1',
  icu: '66.1',
  tz: '2019c',
  unicode: '13.0'
}
```

- env - Lists all environment variables. Better to always read values from a layer above it using `config`.

- release.lts - `node -p "process.release.lts"` will be undefined if not on LTS version of node.

- `exit` event - Will be invoked when the application has nothing else to do or a manual `process.exit` call is made.

```js
process.on("exit", (code) => {
  // do one final synchronous operation
  // before node process terminates.
  // can't stop termination at this point
});
```

- `uncaughtException` event - Will invoke when an error is not caught in the application. If not registered, node will print the stack trace and terminate. When it is registered, node will not terminate and will stay in an unstable state. It's better to forcefully terminate the process when this event is invoked.

```js
process.on("uncaughtException", (err) => {
  // something wnet unhandled
  // Do any cleanup and EXIT forcefully.
});
```
HTML Basics
Published: 2020-04-08 04:13:24

While doing module 2 for my bootcamp at Flatiron school. We have completed many labs where html was needed. HTML may seem like munbo gumbo but it is really straight forward. Here are a few html basics that has helped me through my labs. 

Need to know lists? There are two easy lists you could do which are unordered and ordered lists.
```
<ul>
<li>Corgi</li>
<li>Husky</li>
</ul> 
```
This is an unorder list. You can tell with "ul instead of "ol" which are just sorten for what they mean. "li" basically means list and that you are adding it to the unorder or ordered list. Remember to close your lists with "/ul" or "/ol" and make sure they match what you had started with!
```
<ol>
<li>Corgi</li>
<li>Husky</li>
</ol>
```
Want to add clickable links? Well thats really simple as well you would use the tag "href" here is an example!

```
<a href="Link">Name of the link</a>
```
Here it is in production!

<a href="Google.com">Click here to redirect to google!</a>

a means anchor, and you are referencing the link with href. Afterward, you would have the name or text or the link that users are able to click on!

There are many other basic tags to add words to your website as well. 
```
<h1></h1> - Which is your header 
<p></p> - Which is your paragraph
<br> - to make a new line
```
These are to help you create basic descriptions, title headers, or just having a new line. 

An advanced tag would be making a form! Here is an example

```
<form>
  <label for="dog_name">Dog name:</label><br>
  <input type="text" id="dog_name" name="dog_name"><br>
  <label for="breed">breed:</label><br>
  <input type="text" id="breed" name="breed"><br>
  <input type="submit" name="submit">
</form>
```
Which looks like this in production!
<img src=https://i.imgur.com/E9UT5Ph.png>

These have been the most used html code so far during my labs. It is very helpful to learn the basics to help you get prepared for the more advanced html codes!
Reportes en React
Tags: react
Published: 2020-04-08 03:24:52

Hola. Quisiera saber que herramienta me recomienda para generar reportes en ReactJs , ya sea en el front o en el backend con NodeJs express.....Me gustaria saber si se puede integrar herramientas como JasperReport o algo asi, donde yo haga mi reporte en modo de diseño y luego lo llame desde la aplicacion. Gracias. Mi correo es : loretoruiz@hotmail.com
Access Semantic UI React Dropdown Input Values Using Functional Components
Tags: react
Published: 2020-04-08 02:35:46

I'm using Semantic UI React as the front-end framework for a web application I'm building. The application has a horizontal navigation bar that links to the main pages. Several of the main pages also have sub-pages, so I added a Dropdown component to allow direct access to them. However, I was unable to figure out from the Semantic UI React documentation how to handle the event when an item in the Dropdown is selected.

After some searching, I came across [Saphie Abayomi's article: Accessing input Values from a Semantic UI React Dropdown](https://medium.com/@saphieabayomi/accessing-input-values-from-a-semantic-ui-react-dropdown-f9ede94976fa), which was extremely helpful in pointing me in the right direction. However, she was using class components and my application uses functional components and hooks. So with a little refactoring, I came up with the following:

First, I defined the items to be included in the Dropdown as an array of objects: 
```javascript
const items = [
  { key: "matches", text: "All Matches", value: "/matches"},
  { key: "preferences", text: "Preferences", value: "/matches/preferences" }
]
```

The key can be anything unique, it's there so React can keep track of the elements and avoid needless re-renders. The text is what is rendered in the Dropdown and shown to the user. In this case, the value is the route to the view that should be rendered when the user clicks the item in the dropdown.

The next step is to render the dropdown:

```javascript
  return(
    <Dropdown
      item
      text="Matches"
      options={items}
      onChange={handleDropdown}
    />
  );
```

The key is setting onChange equal to the function that will be called when an item in the Dropdown is selected by the user. The handleDropdown function has two parameters, event and data. The data parameter returns the value from whatever was selected. And without further ado, the handleDropdown function:

```javascript
  const handleDropdown = (event, data) => {
    setRedirectURL(data.value);
    setDoRedirect(true);
  };
```

For the purposes of my web application, redirectURL and doRedirect are both stored in React's Context, so handleDropdown is setting redirectURL to whatever the value of the item selected by the user was and then setting doRedirect to true to trigger the redirect. 

The event parameter isn't particularly useful in my case, so I never read the value. ESLint complains about this, but since I don't have a reason to read it, I don't. Any thoughts on how to solve this minor issue would be appreciated.