teleren/devto huggingface dataset:
How i used a simple python script to compare 2 huge csv file using Pandas
Published: 2021-03-07 10:56:56

Recently i came across a requirement to compare a column data in a csv file with another csv file. These files contains 13 columns with 65 million of rows.

Initially i thought it's simple one and used basic scripting with bash to process line by lines. Then, i realized it took me hours to get that processed. Then i moved to nodejs and then to python. No matter what language i used the result is same. It took more time and i felt it's not worth to process using the regular approach. Then i turned towards pandas library and i tried. To my surprise it processed within 10 seconds.

The requirement is basically to match the emailId in the file1.csv against the md5 email hash of the file2.csv.

Here the snippet i used to compare the matched dataset.


```
import pandas as pd

import hashlib 

def md5hash(mailId):
    print(mailId)
    result = hashlib.md5(mailId)
    return result.digest()

df=pd.read_csv("file1.csv",dtype={ "emailId": "string"},low_memory=False)

# First get the has of the emailId and append to the existing row
df['md5_hashed'] = [hashlib.md5(val.encode()).hexdigest() for val in df['emailId']]
df.to_csv("hashed_records.csv",index=False)


#Store the data in a dataframe
df1=pd.read_csv("file1_with_hashed_email.csv",low_memory=False)
df2=pd.read_csv("file2.csv")
df2_hash=df2["md5_emailId"]

matched=df.loc[df1['md5_hashed'].isin(df2_hash)]
matched.to_csv("non-matched.csv",index=False)

#Just add a ~ to filter the unmatched
unmatched=df.loc[~df['md5_hashed'].isin(df2_hash)]
unmatched.to_csv("non-matched.csv",index=False)

```

I worked with Pandas earlier. But i realized the power of pandas which has saved my hours instead of parsing with nodejs/db based solutions.

Hope this helps someone who faced similar scenarios.
Most Easiest CMS for Designer
Tags: webdev, codenewbie, beginners
Published: 2021-03-07 10:49:06

Hi I am a web designer. I design website and app.
But I know some frontend developer like html, css and some js. I normally use Bootstrap to develop a site.
Recently, I want to learn how I can convert my HTML site to a CMS website so that I can update blog in the site.
I wonder which is easiest for me to learn since I only have these very basic skill.
Netlify, Twill, Craft CMS, Statamic, Grav << which is better and easiest for me to learn?
Thank you.
#1 Creating basic express server and react client and the use of concurrently
Tags: javascript, express, react, 100daysofcode
Published: 2021-03-07 10:33:10

#Server
1) Create a project directory
```
mkdir basics
```

1) Initialize the project using npm init.
```
cd basics
npm init
``` 

2) Since we'll be using express as our backend, let's install it using npm. Also, since we'll have our client and server in the same project folder, let's install concurrently too. We'll use it later though.
```
npm install express concurrently
```

3) Create a server directory in project root folder and then create a server.js inside it. Add the below code in server.js to do basic server setup.
```
const express=require('express');
const app=express();

const PORT=process.env.PORT || 5000;

app.listen(PORT, ()=>{
    console.log(`Server is running on ${PORT}` )
})
```

4) Verify the server setup.
```
node ./server/server.js
```

5) Now let's get to a bit advanced stuff, not advanced in the real sense but since we're talking from a beginners prespective, everytime, you do any changes to your server, you need to restart it, so let's install nodemon to get rid of that.
```
npm install nodemon
```
One important thing to note here is if nodemon is not installed globally, the below command may throw an error.
```
nodemon ./server/server.js
zsh: command not found: nodemon
```
The reason is that you need to install it globally by using the command
```
su npm install --g nodemon
```
else you may use the npx keyword to run it.
```
npx nodemon ./server/server.js
```
#Client

1) Create a client directory in the project root
```
mkdir client
```

2) Navigate to the client directory and create a react application by using the below command. Note that, since we don't want to create a new folder inside client folder, we use dot operator.
```
npx create-react-app .
```
You may test your client by running the below command
```
npm start
```
##Refactoring 
If you notice, you have three different commands to start you server and client.
```
node ./server/server.js
nodemon ./server/server.js
npm start 
```
Let's just add all these commands within the scripts of package.json. So, let's open package.json and add the below
```
"start": "node ./server/server.js",
"server": "nodemon ./server/server.js",
"client": "npm start --prefix client",
```
Now let's make use of concurrent package we installed earlier. Add a dev script inside scripts and add the below content.
```
"dev": "concurrently \"npm run server\" \"npm run client\"",
```
Now stop all previously run server and start both server and client using
```
npm run dev
```
Now, since my client is running on http://localhost:3000 and server on http://localhost:5000 , let's add a proxy in the client side so that everytime it makes a request to server, we don't have to explicitly type the whole url. Open package.json inside the client folder and add 
```
"proxy":"http://localhost/5000"
```

This is my first blog people, do leave your feedback and any improvement areas if need. I would love to hear from you.
Extracting Audio from Video Clips using Python
Tags: programming, python, datascience
Published: 2021-03-07 10:05:08

Hello World!

This mini blog is about how one can extract audio files from video clips using Python! This can be done in less than 5 lines of code but what is more important is to get started with **Moviepy** module.

We are going to take a look at **Moviepy** module of python which is widely used for editing videos using python.
Moviepy makes video editing painless in python and by using just few python one-liners one can get desired outcome!
Here we are going to extract audio file from a sample video file.
I have used a sample MP4 video clip in this blog. You can use any video file.

Let's get into the coding part!

First of all we need to install moviepy module in python by typing the following command in terminal window:
```python
pip install moviepy
```
The next step is to install **FFmpeg** module. FFmpeg is a multimedia framework used for handling audio, video and other streams.
To install **FFmpeg**, type the following command in the terminal window:
```python
pip install ffmpeg
```
If the installation was successful then you are all set up!

We will begin by importing the moviepy module:
```python
import moviepy.editor as mp
```
The next step is to copy the sample video in the project folder and  define the video file in python code. **Moviepy** comes with a handy `VideoFileClip()` method that imports the sample video as a moviepy object. Here is how you can import the video file inside the python code:
```python
video = mp.VideoFileClip(r"sample.mp4") #here sample.mp4 is the name of video clip. 'r' indicates that we are reading a file
```
Now it's time to convert the video file into audio file. We will be converting the .mp4 file into .mp3 file. To do this add the following line in the program:
```python
video.audio.write_audiofile(r"output.mp3")  #Here 'output.mp3' is the name of the audio file
```
You are all done!
The final code will look like this:
```python
import moviepy.editor as mp

video = mp.VideoFileClip(r"sample.mp4")
video.audio.write_audiofile(r"output.mp3")
```
Run the program and wait for a while. You may see the progress in the terminal window
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7e5j3b91pl0x463979j5.png)

And that's it! 🎉 Check the project folder and you will see the `output.mp3` file which has been extracted from the sample video file.

This was the first blog post about Moviepy. I will be creating more projects using it and will write blogs about them as well!
Stay tuned!

If you are stuck at any point, do let me know in the comments section or ping me in our [discord server](https://discord.gg/ggvtTje) where 200+ programmers are hacking on something!
CSS Responsive Profile Cards with Slider
Tags: responsivecards, profilecards, cardslider, owecrousel
Published: 2021-03-07 09:27:11

Hello readers, today you learn how to create CSS Responsive Profile Cards with Slider by using only HTML & CSS, many people are using JavaScript and Plugins to make card slide but in this programming, we will make beautiful profile cards with sliding animation in HTML & CSS. earlier I have shared <a href="https://www.codinglabweb.com/2020/12/responsive-profile-cards-using-html-css.html">Responsive Profile Card without sliding animation</a> but now we will make responsive profile cards with sliding feature.

Profile Card means the identity card of a particular person that contains some details of them like name, job position, and some other information. On the webpages, we can see various people's profile cards with different designs. The main motive to append profile cards on the webpage is to make users contact the person easily.

Here is an image of the profile cards that I have uploaded on the webpage. Basically, profile cards look like this. As you can see there is a different image on the top and person's name their job and some media icon to contact them through social media. At the bottom of those cards, there are two buttons, that left side button is active so its width increase, when we clicked on the second button those three profile cards moves left side and another three cards appear smoothly.

#You Might Like This:#
<a href="https://www.codinglabweb.com/2020/12/profile-card-design-using-html-css.html">CSS Animated Login Form</a>
<a href="https://www.codinglabweb.com/2020/12/profile-card-design-using-html-css.html">Responsive Login Form Form</a>
<a href="https://www.codinglabweb.com/2021/02/z-index-transition-on-image-using-css.html">Responsive Registration Form</a>
<a href="https://www.codinglabweb.com/2020/12/product-card-design-using-html-css.html">Login Form WIth Social Media Icon</a>

#Source Code#
You can download all source code from the given link. <a href="https://www.codinglabweb.com/2021/03/css-responsive-profile-cards-with-slider.html">Click Here To Download All Source Code</a>
Android Studio | opened an existing project with no windows to work on
Tags: applications, android, studio, app
Published: 2021-03-07 09:19:14

Hello, application programmers,
My question is about Android Studio.
I opened an existing project.
And it's opened without windows or tools. (why? - How to fix it?)
Here is an image if you want to check,
https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xqh1oyez7pec0r8tvzli.png
Also, there is a selection I don't know why it appears which is (compare with previous apk)
CI/CD: continually delivering better codes
Tags: cicd, integration, deployment, devops
Published: 2021-03-07 09:11:29

At the time of writing this post, I am still not done reading this special edition of 'Entwickler Magazin', featuring my favorite topic: CI/CD (continuous integration / continuous deployment).

These are the points from that magazine that I noted for myself.

## DevOps models:
1. One-team model: the Ops team is a subset, a part of Dev team
2. Collaboration model: DevOps is achieved in the collaboration betwen Dev team and Ops Team.
3. IAAS-model: infrastructure as a service.

## CI/CD:
1. Tools:
 * Jenkins
 * Harness
 * GitLab
 * GitHub
 * Atlassian Bamboo

2. CI tools: are to enable code components run together.
3. CD tools: are for automated tests and deployment.

## Requirements for CD in an enterprise:
Abstract: CI != CD
1. Continuous integration: automated tests and analysis.
2. Continuous delivery: delivery pipelines
 * scripted
 * versioned
 * reproducable
3. Feature toggles
4. Zero downtime
5. Blue Green Deployment:
 * 2 complete Production environments
6. Canary Development
7. Secrets management, e.g. Vault:
 * query via API - no clear/plain text leaves the vault.
8. Security and compliance - especially for containerized microservices:
 * container security
 * interservice communication

// To be continued.
🧽 Clean fish shell command history from typos automatically
Tags: fish, shell, plugin, fisher
Published: 2021-03-07 09:00:11

Don't you hate it when you spend half an hour searching for a command through your shell history (you know it's there, you entered it two months ago yourself!), you finally find it and happily press `Enter` only to realize two seconds later... it's not that one, it's another one you'd entered just before that, the one with the _typo_. Ugh. Awful. Ungodly. Unproductive. Your day is ruined... unless you use _Sponge_.

That's it. Sponge is a little plugin for the [fish shell](http://fishshell.com/) that runs in the background and keeps your history clean from typos, incorrectly used commands, and something you don't want to store due to privacy reasons.

Check out Sponge on [GitHub](https://github.com/andreiborisov/sponge) and feel free to make suggestions or raise an issue!
Can I Have Multiple SPF Records on My Domain?
Tags: email, spf
Published: 2021-03-07 08:53:32

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/o470rrfku7lg76aibhan.png)

Can you have multiple SPF records on a single domain? The answer is no: a domain **MUST NOT** have multiple SPF records, otherwise SPF fails with PermError.

An SPF record is a TXT record in the DNS starting exactly with "v=spf1", followed by an array of mechanisms and/or modifiers.

An SPF check starts by fetching all TXT records starting exactly with "v=spf1" on a domain:
* if no such record is found, it returns ```None```;

* if multiple such records are found, it returns ```PermError```.

For example, if you have 2 TXT records on yourdomain.com:

| Record Type   |  Name            |  Value         |  TTL    |
|---------------|:-----------------|:---------------|:---------|
| TXT           |  yourdomain.com  | v=spf1 include:_spf.google.com -all | Default |
| TXT           |  yourdomain.com  | v=spf1 include:servers.mcsv.net -all| Default |

All emails sent on behalf of yourdomain.com will fail SPF authentication with PermError.

In order to fix this, you need to add all the mechanisms that contain legitimate IP addresses in 1 SPF record, as shown below:

| Record Type   |  Name            |  Value         |  TTL     |
|---------------|:-----------------|:---------------|:---------|
| TXT           |  yourdomain.com | v=spf1 include:_spf.google.com include:servers.mcsv.net -all | Default |

For SPF authentication to pass, all these conditions must be met:
* you have exactly 1 SPF record on your domain;
* the record's syntax is correct;
* [the total number of DNS lookups can't exceed 10](../spf-permerror-too-many-dns-lookups-when-spf-record-exceeds-10-dns-lookup-limit). Otherwise, use [Safe SPF](https://dmarcly.com/safe-spf) to fix it.

#### Monitoring SPF records

Now that the validity of SPF settings is crucial to SPF authentication success, you might want to receive alerts when an SPF setting is messed up, so that you can take action to fix it as soon as possible.

DMARCLY's domain monitoring feature allows you to do exactly that. In order to monitor your domain:
1. [log in](https://dmarcly.com/login) to the dashboard;
2. go to DNS Records => Domains, add your domain;
3. go to Reporting and Alerts => DNS Alerts, create a recipient with your email.

From now on, if multiple SPF records are found on your domain, you will get an email notification that looks like below. Nice!

![](multiple-spf-records-on-domain-warning.png)

Related questions:

[Can I Have Multiple DMARC Records on My Domain?](https://dmarcly.com/blog/can-i-have-multiple-dmarc-records-on-my-domain)
[Can I Have Multiple DKIM Records on My Domain?](https://dmarcly.com/blog/can-i-have-multiple-dmarc-records-on-my-domain)

[Original post](https://dmarcly.com/blog/can-i-have-multiple-spf-records-on-my-domain)
Web Development Path 
Tags: webdev, html
Published: 2021-03-07 08:28:20

There are numerous ways you can approach getting the right stuff you need to turn into a designer. 

There are direct educational plans that encourage you a touch of everything - like freeCodeCamp's full stack advancement educational plan. 

And afterward there are gigantic arrangements of abilities - and in some cases arrangements of assets - you can use to "pick your own experience" through the learning cycle. 

Outstanding amongst other such rundown of abilities is Kamran Ahmed's Developer Roadmap, which he routinely refreshes. 

This is no customary rundown of assets. Kamran has meticulously delineated every one of the abilities he feels are fundamental for breaking into various fields of improvement.

Read also:
 [How to validate an email address in JavaScript](https://tutorialscamp.com/how-to-validate-an-email-address-in-javascript/)
[JavaScript Vs Jquery: Which Is Better?](https://tutorialscamp.com/javascript-vs-jquery/)
 [Java vs JavaScript](https://tutorialscamp.com/java-vs-javascript/)
[Angular 10 tutorials for beginners](https://tutorialscamp.com/angular-10-tutorials-introduction-for-beginners/)
 [Shape builder tool techniques in illustrator](https://tutorialscamp.com/illustrator-shape-builder-tool/)
 [JavaScript vs python Difference](https://tutorialscamp.com/javascript-vs-python/)
Python and the Gambler's Fallacy
Published: 2021-03-07 08:25:20

I'm not sure what the etiquette is here for sharing posts. I hope a link to my blog is OK. I reckon getting the p5.js code to work here could be tricky...

Explore the Gambler's Fallacy with Python and p5.js by simulating sequences of coin flips.

https://compucademy.net/the-gamblers-fallacy/

\#python \#STEM \#compsci \#computermaths \#thinking
In-depth document comparison 
Tags: programming, productivity, webdev, csharp
Published: 2021-03-07 07:36:10

Do you want to compare documents with all small details? Have a look at the following .NET code:

```csharp
Comparer comparer = new .Comparer(sourcePath);
comparer.Add(targetPath);
CompareOptions compareOptions = new CompareOptions();
StyleSettings changedStyleSettings = new StyleSettings();
changedStyleSettings.HighlightColor = Color.Red;
compareOptions.ChangedItemStyle = changedStyleSettings;
comparer.Compare(@"D:\result.docx", compareOptions);
```
This is not it. This document comparison API provides a lot of comparison options and support for multiple [file formats](https://docs.groupdocs.com/comparison/net/supported-document-formats/). The changes detection algorithms used by [GroupDocs.Comparison](https://docs.groupdocs.com/comparison/net/groupdocs-comparison-overview/) allows to detect changes in different document parts and blocks.
From style changes (e.g. italic, bold, highlighted text) to the item insertion or deletion, everything will be compared.
Have a look at the following screenshot:
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/t1k2wmnk9f401tg1opgt.png)
March LeetCoding Challenge 2021 — Day 5: Average of Levels in Binary Tree
Tags: programming, codenewbie, computerscience
Published: 2021-03-07 07:03:55


Today, we will solve the 5th problem of the March LeetCoding Challenge.


## Problem Statement

Given a non-empty binary tree, return the average value of the nodes on each level in the form of an array.

**Example 1:**

    **Input:**
        3
       / \
      9  20
        /  \
       15   7
    **Output:** [3, 14.5, 11]
    **Explanation:**
    The average value of nodes on level 0 is 3,  on level 1 is 14.5, and on level 2 is 11. Hence return [3, 14.5, 11].

## Solution

According to the question, we have to find the average of each level in a binary tree. We can think about this problem as a Level Order Traversal problem. Read more about Level Order Traversal [here](https://www.geeksforgeeks.org/level-order-tree-traversal/).

Let's try to visualize the problem. So when we do the level order traversal, we will find the average of each level. This seems simple right? Let’s go through the code once.

![Average Level Sum in Binary Tree](https://cdn-images-1.medium.com/max/2560/1*HuKg9cdHmdHzZGUMGtP_bw.png)

We have levels list to keep the level sums. We use a queue to-do the Level-Order-Traversal. We first check the size of the queue, this size denotes the number of nodes in a particular level of a binary tree. We add all the values of the nodes and find their average and store it in the list. At last, we return the list.
{% gist https://gist.github.com/sksaikia/4cd760cfb4db8d0eea48e5c4a525a748.js %}

Check out the repository.
{% github sksaikia/LeetCode %}
Using react-leaflet to generate one tiled layer
Tags: leaflet, tutorial, geoserver, dataviz
Published: 2021-03-07 05:58:10

The problem: Leaflet does not support single tile layer.
==
*(This was posted in 2017, on medium)*

Which means that if you need to render some dynamic cluster, generated by geoserver, like this:![leaflet-1](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5oo1lcu8nuax022sw2nd.png)

Leaflet will, instead, show something like this:![leaflet-2](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0cyfmejmssuk0sxqotlt.png)
 
Which is messed up. Fortunately, we (me and [Arlindo Pereira](https://dev.to/nighto), at MPRJ) got a workaround using [ImageOverlay](http://leafletjs.com/reference-1.2.0.html#imageoverlay) element from [Leaflet](http://leafletjs.com/reference-1.2.0.html), which is pretty simple, but it took a lot of time to discover.

[React-leaflet](https://github.com/PaulLeCam/react-leaflet) is a lib that exposes components to work with leaflet. So it exposes react components like Map, TileLayer, WMSTileLayer, and so on. It also exposes ImageOverlay. To get this to work, you need to use ImageOverlay component like this:

```
<ImageOverlay
  bounds={imageBounds}
  url={imageURL}
  transparent={true}
/>
``` 

Where imageBounds needs to be a Leaflet Bounds object, which should be the bounds for the current visible portion of the map, like this:

```
{
 “_southWest”:
 {
 “lat”:-24.27701247166408,
 ”lng”:-46.65893554687501}
 }
 ”_northEast”:
 {
 “lat”:-21.927759064052037,
 ”lng”:-37.86987304687501
 }
}
```

The imageURL parameter is the URL to the service that will generate the image and should be like this:

```
let imageURL = `${ENDPOINT}?
SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&FORMAT=
image%2Fpng&TRANSPARENT=true&STYLES&LAYERS=
${layer.layerName}&SRS=EPSG%3A4326&WIDTH=
${clientWidth}&HEIGHT=
${clientHeight}&BBOX=
${imageBounds._southWest.lng}%2C
${imageBounds._southWest.lat}%2C
${imageBounds._northEast.lng}%2C
${imageBounds._northEast.lat}`
```

This is a WMS call. The main things you should note are the Height, Width and Bbox attributes. The height and width need to be in pixels and related to the size of the map in your app, so to speak. Our map fits the html document, so we got it like this:

```
let clientHeight = document.body.clientHeight
let clientWidth = document.body.clientWidth
```

And the Bbox is the same coordinate as mentioned before.
You can see how it was before:

![mprj-1](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fini49odyrwxajp58ea8.png)

And how it is now:

![mprj-2](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7p9lumi2qcyu8y1lvz3w.png)

Or here, live: [http://apps.mprj.mp.br/sistema/inloco/](http://apps.mprj.mp.br/sistema/inloco/)
Here to see the code: [https://github.com/MinisterioPublicoRJ/inLoco-2.0](https://github.com/MinisterioPublicoRJ/inLoco-2.0)

InLoco is aGeographic Information System (GIS) used by Ministério Público do Estado do Rio de Janeiro to show social, institutional and administrative data, made with React and Leaflet, interacting with a public GeoServer backend. You are welcome to send you PR =)