teleren/devto huggingface dataset:
Need help python text matching
Tags: help, python
Published: 2020-07-09 23:25:05

Hi DEV community!

I have a project idea that I would like to dive into but I need your help defining some major points.

The project is about matching strings to the most similar string in a different dataset(tables) in Python.

For example:
Table 1
**name**/**city, country**
*Mark Smith*/*New York City, USA*
*Mirko Smirk*/*New York City, USA*
*John E. Doe*/*Paris, USA*
*Jane Doe*/*Paris, France*

Table 2
**name**/**city, country**
*Mirko S.*/*NYC, US*
*Mark S.*/*NYC, US*
*Jane D*/*Paris, France*
*J. Doe*/*Paris, US*


The idea is to match each person with itself on the other table.
For example row 1 of the first table would correctly match with row 2 of the second table.

As you can see there are no perfect matches and sometimes it may match the first column but not the second. The second column should have more weight in the decision.

I first thought of an approach based on **Levenshtein Distance** to calculate the differences between sequences using FuzzyWuzzy package. But I encountered many issues like the following:
**Table 1**: 
*Franklin Delano Roosevelt*
*Franklin Da Turtle*

**Table 2**: 
*FDR*

Levenshtein Distance would say that *Franklin Da Turtle* is a better match than *Franklin Delano Roosevelt* for *FDR* since it has fewer characters to compare.


In real-life data, I will have many columns to validate the accuracy of the decision but I'm stuck anyway.

Is there a better approach? I first thought of some vector usage of NLP but isn't my area of expertise.

Any ideas will be well received. 

Leave your comment if you know something better than FuzzyWuzzy approach.
Journey-ing With C
Tags: computerscience, learning, c, codenewbie
Published: 2020-07-09 23:17:51

# Preparing

Textbook Used: *C Programming: A Modern Approach* by K. King

![img](https://images-na.ssl-images-amazon.com/images/I/41+bFKtFHjL._SX405_BO1,204,203,200_.jpg)

## Textbook or Online Materials?

*C Programming: A Modern Approach* is a go-to textbook to learning C. I decided to use a book instead of searching for tutorials online. I thought that using a textbook would give me structure in self-teaching myself C instead of looking for online posts. 

Unlike online tutorials, they go through a lot of drafting, editing and peer-reviewing before publication. And usually, the ones making them, are experts on the matter. That being said, it doesn't hurt to use google to help you once in a while.

## Have a Mentor (a.k.a someone you could ask for help)

I was really lucky because my professor was willing to teach me (I'm his experiment for online teaching this fall) BUT, if you don't know where to look, here are some suggestions (unordered):

1. Discord
2. Slack
3. Reddit
4. Learn with a friend(s)
5. Just wing it

Those are where I would start looking. If you don't want to and want to do this alone because it's too much work, try learning with a friend (I've never actually tried this because...). 

*C Programming: A Modern Approach*, is a really good book, the chapters aren't too long and lots of examples and practice work so, yes! Less falling asleep.

## Which Operating System (OS) Will Be Used To Learn?

All right, so the thing with C is that it's really easy to set up in an Ubuntu-based OS (I'm using Linux). SO I set up a **Virtual Machine with the operating system Linux**. 
This is the part where a **mentor or having someone to ask is a major benefit**. I had absolutely no idea (and still now) how to confidently set up a Virtual Machine. 🙃

The benefits of using a virtual machine:

1.  Not having to worry about corrupting your current OS files and just generally messing up your OS.
2. With a Ubuntu or Linux OS, you can gain experience in using ***bash\*** the language used for the terminal in Ubuntu/Linux OS's
3. Get experience using the ***man pages\***, which is basically an offline mode of documentation about C libraries and functions
4. More organized since (most likely) all you'll be doing is doing C in there.

## Motivate Yourself

This is why I also recommend to learn or be taught by/with someone because it will motivate you to just do it. 

My main motivators are:

1. Knowing the language before enrolling in the class will help me understand the language deeper when it comes time for my class. 
2. Also, it would be another tool under my belt.
3. And plus, it's fun. 😄
How I built Ngrok Alternative
Tags: python, go, sideprojects, javascript
Published: 2020-07-09 23:12:32

<b>Here's a quick intro of what I built<b>
{% youtube BXuB3cK8R0g %}

<a href="http://ngrok.io">Ngrok</a> is a fantastic tool that helps developers to expose their localhost to the Internet with minimal effort. One day I was going to share a local project with a client without deploying it somewhere.  But this time I needed to expose <b>2 ports</b>: one running frontend and the other backend of the project, I found out that the ngrok's free plan only allows one tunnel at a time. Later that day, I also found out that there's a <b>40-requests/minute limit</b>.

<b>Bam! An idea.</b>
What if I build an alternative instead of paying $5/mo.? 
That's what developers do, right? I also had the idea of making it open-source.

<b>It's time to plan.</b>
There are obviously 3 parts:
  - a developer with a command-line tool (cli)
  - a proxy server
  - a client with a browser. 

I was sure there had to be at least those steps to make it work:
  1. a developer connects to the server via a cli with a port number: `jprq 8000`
  2. the server responds with an assigned domain. `amazing-coder.jprq.live`
  3. a client opens the domain in a browser: `https://amazing-coder.jprq.live`
  4. the server receives the request and sends it to the cli.
  5. the cli makes the request to localhost at the given port and sends the response to the server.
  7. the server responds back to the client with the response it received from the cli.

Here's a chart to help you picture those steps.
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/ldpnnlez5fnv0g4r2ty3.jpg)

The cli and the server needs to make back and forth communication. A carefully managed TCP Sockets would do a great job but would take a long time to implement. 
I decided to use and used the WebSocket Protocol. As you might know, WebSocket is a hop-by-hop protocol and sadly, my ngrok alternative now can't support Polling and HTTP streaming (because they never end or last too long).  

The command-line tool is written in Python and published to PyPy. Currently implementing the command-line in javascript to publish in npm just for fun. And the server-side is in Golang. I think it was the best decision to choose golang because of its easy data sharing between goroutines and learned about leaking goroutines the hard way. I now have a good understanding of memory leaks because of this project.

Another small but impactful mistake I made during implementing the client-server communication was using JSON. I realized it only after adding a feature for handling files. With JSON one can only serialize strings. To turn file contents, bytes into a string, I needed to Base64 them. It turns out to be CPU intensive process. It's better to use BSON, I believe.

The project is open-source and waiting for your contribution. Take some time to visit the GitHub repo at <a href="https://github.com/azimjohn/jprq">github.com/azimjohn/jprq</a>

Keep building,
Cheers.
How do you feel about Neuralink?
Tags: news, discuss
Published: 2020-07-09 23:03:49

> The last major update from Neuralink came roughly a year ago, when it shared that it will be using a surgical robot to implant gossamer-thin wires into a person’s brain, connected to an external computer processing unit, and that ultimately it hopes to make the connection wireless between the two for maximum freedom and flexibility.

> Neuralink revealed in July 2019 that it had already performed successful tests of its technology on mice and even apes, and that it would be pursuing testing on its first human subjects starting as early as the following year — which is this year, 2020, if you’re keeping track.

### 👉 [TechCrunch: Elon Musk sets update on brain-computer interface company Neuralink for August 28](https://techcrunch.com/2020/07/09/elon-musk-sets-update-on-brain-computer-interface-company-neuralink-for-august-28/)

How do you feel about all of this?
How do you put data for other people to use?
Tags: discuss, help
Published: 2020-07-09 22:41:18

Suppose you have scraped data from a site in `JSON` format which can be also used by other people. How do you expose it? 
Do you make a url directly give out that `JSON` data? Or there is a better way. I was thinking of something like a GraphQl server. What do you guys think?
Testing Legacy Vue Projects - The right mindset  Part 1 of 2
Tags: vue, javascript, testing, tdd
Published: 2020-07-09 22:37:01

Is the joy of your really fancy component library or project short-lived by the fear that it lacks testing hence it is fragile? Missing out of TDD isn't an eternal curse. You can salvage the situation. Lemme show you how.  This is part of a series of three(3) articles, first two articles will prepare you to have the right mindset towards this task. And the final article will feature some real examples.


![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/sv8c1nq6bz3avxjgeob8.jpg)


**NO! You don't need to rewrite the project.**

>"The other developers got it wrong". I will destroy this temple and build it up in 3-days.

The first thought that comes to the minds of noobs is the plan to begin a total rewrite of legacy projects. This consideration may be trivial for small projects, however, for large projects the mere thought of this marks the end to the adoption of testing.

**Imagine your legacy project in terms of modules and components.**

You might be very fortunate if your existing project is modular or is broken down into small components. But don't worry if it isn't, cos you probably just got your second shot at TDD. Your future components will be test compliant.

**Begin 'decomposing' large Vue components into smaller components**

>Everything matters. Appreciate your little gains. Every step you make towards your ultimate goal of having your code tested counts. 

Evaluate your large Vue components, break them into smaller components. *DON'T be in a haste to decommission large components or refactor them*. Take your time to understand how several pieces communicate with themselves.

In the next piece, I will cover some crucial tips that will help you create better re-usable and tested components.

Let me know your thoughts...
JupyterLab in a virtual environment
Tags: python, jupyter
Published: 2020-07-09 16:05:46

## Why?
🏋️‍♀️ When it comes to running JupyterLab & JupyterNotebooks, the install is pretty hefty, with quite a few dependencies.
🦉 To avoid the travails of picking through a messy base install of Python on your dev machine, it is wise to always use a virtual environment.
👍 Thus running JupyterLab in a virtual environment is a *good idea*.

## How?

Here's how using macOS.

1. Navigate to a suitable directory in which want to work, or make a new one.
2. Copy the Jupyter Notebook that you want to work on to this directory.
3. Create a virtual environment and give it a name, in the example below, I'm using 'jup1'. 
    ```bash
    python3 -m venv jup1
    ```


4. Activate the the virtual environment.  

    ```bash
    source jup1/bin/activate
    ```


5. Now that the venv is active we can install JupyterLab.  

    ```bash
    pip install jupyterlab
    ```

6. Start JupyterLab.  

    ```bash
    jupyter lab
    ```

## Extra Tips
* The shortcuts for Jupyter Notebooks really help to speed up working with them. It pays to invest some time in learning them. Or, if you can't be bothered with that, just remember that 'shift' + 'return' runs a cell.
* If you want to check your notebooks into version control, such as github, make sure you clear the outputs first. To do so right-click on the notebook, or Edit > Clear All Outputs. Then save and do your commits.
Team Management: Who Has The Final Say?
Tags: leadership, management, project
Published: 2020-07-09 22:35:16

It's critical that you understand on your team who has the final say. Here are some questions. Is the team leader's authority clear? I've worked on teams where the team leader didn't have the authority. 

The way I framed that is the team leader didn't have any teeth. They could make certain decisions, but whether anyone actually respected those decisions or followed through with those decisions was questionable. 

They might have had authority on paper, but people weren't following them. What if the team leader gives or bestows the authority to other team members. In other words, they choose a few people on the team who can actually make final decisions on their behalf. 

What about when your team meets with other teams? 

Who has the final say? 

Do the other teams have more power than your team? 

If the team leader, who has the authority, doesn't go to those meetings… is it clear that the people who have gone in his or her place have the same authority to make a final decision that the team leader has? 

Going into politics, could it be that someone outside of the team is making the decisions? This could be good because people outside of the team might have more authority at the corporate level, but if they're not involved in the discussions and understand what the problems are, they might make some very poor decisions.
Day 171: Bad
Tags: hiphop, code, coding, lifelongdev
Published: 2020-07-09 22:23:27

_liner notes_:
- Professional : Got to catch up with a co-worker that lives in the UK. Interesting to see how others are handling everything that's happening. Had our wider team meeting which had time for people to demo what they've been working on. People are working on some really cool things. I got to demo 2 projects I've been working on. I showed the Web Component I built and also the web application to help with livestreaming. Everything worked so that's a plus! haha Afterwards, I got to show another coworker that live streams how it may be able to help her. We got it working and her CPU usage was like 2% and she said it normally goes to 30% because of a program she was using that could be replaced with the web application. That was cool. Hopefully it can help. For the rest of the day, fixed some stuff with the pull request of my Web Component I submitted to the group repo. Finally passed all the checks! haha Then I worked on my blog post. I'm really bad at writing blog posts. haha.

- Personal : Got the screen sharing component of my side project working! Pretty cool. I'll be able to use it for both the host and guest with different features for each. Also, went through a bunch of tracks for this week's radio show and submitted last week's show to the station. Oh and I watched the final episode of "DoroHeDoro" anime on Netflix. Ended on a cliffhanger. Looking forward to another season.

![View of the oceanside cliffs in County Clare, Ireland](https://dev-to-uploads.s3.amazonaws.com/i/ecaq1gu1uvskta4e4ymx.png)

Going to work a little more on my blog post. Make some more components for my side project. Listen to more tracks. Maybe an anime episode. Then call it a night.

Have a great night!

peace piece
Dwane / conshus
https://dwane.io / https://HIPHOPandCODE.com
{% youtube Sd4SJVsTulc %}
#032 Kubernetes - Pods lab 2
Tags: kubernetes, devops, tutorial
Published: 2020-07-09 22:13:14

# Introduction

this is part 32 from the journey it's a long journey(360 day) so go please check previous parts , and if you need to walk in the journey with me please make sure to follow because I may post more than once in 1 Day but surely I will post daily at least one 😍.

And I will cover lot of tools as we move on.
***

# lab
in last part we create a pod . we are going to delete this pod using
![delete-pod](https://dev-to-uploads.s3.amazonaws.com/i/89p3adpchw72qconlmsg.png)
```bash
kubectl delete first-pod
```
As we see the pod got deleted , now I am going to use another random container from docker hub , I found an container called simple-api you can take a look at it [here](https://hub.docker.com/r/emondek/simple-api)

let's create a pod from it
![create](https://dev-to-uploads.s3.amazonaws.com/i/ysp4l7ikcvr9esss2egb.png)
```bash
kubectl run first-pod --image=emondek/simple-api:latest --restart=Never
```
let's take a look
![get-pods](https://dev-to-uploads.s3.amazonaws.com/i/u2q55si87zi1199bi4ws.png)
```bash
kubectl get pods
```
we can see status creating container , if we need more info about it we type
![describe](https://dev-to-uploads.s3.amazonaws.com/i/hlm9jwkjm02951mwwmfy.png)
```bash
kubectl describe pods/first-pod
```
if we scroll down we can see events
![events](https://dev-to-uploads.s3.amazonaws.com/i/c0w0nfjan6ati9ux5de2.png)
it's now on image pulling phase

after pulling we can see now that status is RUNNING
![done](https://dev-to-uploads.s3.amazonaws.com/i/aikzhm5ack5fjxs5pbb3.png)

now to run this pod in browser we need services(will talk about them later) but we have a nice hack to run it for now

![forward](https://dev-to-uploads.s3.amazonaws.com/i/f1di2lbjpt2kbyp4xv06.png)

```bash
kubectl port-forward pods/first-pod 8080 
```
8080 is the port the we decide to use in dockerfile 
![8080](https://dev-to-uploads.s3.amazonaws.com/i/ptu8s0czqc13dhqwajep.png)

if we go to the browser and type [http://localhost:8080/](http://localhost:8080/)

we have our pod running 

![running](https://dev-to-uploads.s3.amazonaws.com/i/3twjvxlc17y09clx47jh.png)

we can also access this pod using
![access](https://dev-to-uploads.s3.amazonaws.com/i/uisobmm82q66vde0lc3e.png)
```bash
kubectl exec -it first-pod -- bash
```
We use -- to separate the commands that related to kubectl and linux.
And every change we make to this pod will be discarded just like container so also we need volumes here :D
Monitoring for Startups. Which affordable solutions are you all using?
Tags: discuss, monitoring, startup
Published: 2020-07-09 22:01:55

Often, or at least to what I've been exposed to, the Startups have a reduced budget for infrastructure and they are in constant delivery pressure.

This two constant implies that you have some issues in production that you might no realise easily that they are happening.

I would say, that when the Startup is an early stage, the website traffic is still low. So we don't need a huge infrastructure support a lot of logs per minute.

---

That's when _Affordable Monitoring_ comes into play. I would define _Affortable Monitoring_ to a small budget solution is at least able to:

- Centralized Logging with Search capabilities
  - For Backend, Frontend and Mobile
  - At least 7 days retention time
- Alerts when the service is down or in degraded service state
- Some metrics like:
  - Request Per Second
  - 2XX, 4XX, 5XX errors in a period of time
  - Mean Response Time

We should also define what a small budget means. I would say 20-30EUR/month for all the services, not per host.

---

Here what we are using :smile: ↓

We have been using [CloudWatch](https://aws.amazon.com/cloudwatch/) because we are using AWS as a Cloud Solution, and most of the requirements are in the free tier so it's a cheap solution for us.

It's true that it doesn't support, as far as I know, frontend logs. That's why we were considering [Rollbar](https://rollbar.com/) for the React WebApp.

And you? What do you propose as a good stack with those constraints? Would you adopt [Grafana](https://grafana.com/), [Prometheus](https://prometheus.io/) that early? What about [ELK Stack](https://www.elastic.co/)? Is it worth compared with CloudWatch? Self-managed or third party hosted? Which metrics do you monitor?
Apple Silicon Is Also A Chance To Improve Security
Tags: security, microsoft, apple
Published: 2020-07-09 21:18:17

[Toward trusted sensing for the cloud: Introducing Project Freta - Microsoft Research](https://www.microsoft.com/en-us/research/blog/toward-trusted-sensing-for-the-cloud-introducing-project-freta/)

> The fact that sensor evasion is possible at all is surprising to many outsiders to the field of computer security. Anyone who’s visited a retail store understands that it’s a good idea to put the security cameras out of reach—why haven’t we done something similar in the cloud? The answer is a familiar one: backwards compatibility. Our first networked computers didn’t have the silicon real estate to devote to an isolated security sensor. Opportunities for technology companies to break backwards compatibility and “greenfield” redesign with security in mind have appeared only periodically, seen primarily in the mobile and console industries. These redesigns have allowed for increasing hardware separation between the compute plane and the security plane

Apple changing Macs to Apple silicon presents a similar opportunity to break backwards compatibility and redesign the security architecture. They kind of tried it with the T2 co-processor, but the complete transition may lead to a something more. How the isolation (lock-down) in macOS has been progressing up to Catalina might have also been in preparation for this, and Big Sur might be closer to iOS in more ways than the UI. I look forward to all the analyses that might come out from Objective-See.

I suppose some people get annoyed at the degree of lock-down Apple has been introducing. Personally, I've locked down my Linux instances even more. I just don't have the expertise and time to figure out how to lock it down without impairing the user experience too much, and I am glad Apple has been at it all this time.
What is a Load Balancer?
Tags: beginners, nginx, backend, loadbalancer
Published: 2020-07-09 21:18:14

# Why? 
---
imagine that you've worked on a startup website or a web app for a client and it started to get 100 visits per month. one year in, and the number of visits drastically increased to the point that the client called you complaining about the app's performance, so being the great engineer you are, you decided to upgrade the server's specs hoping it will solve the response time problem and it worked! 

But what if the server hits 3000 visits per minute and costumers started getting a  "502 bad gateway" and upgrading the server specs isn't an option being too expensive or that you are already using the maxed out specs the hosting service could offer.
* Hosting services list :
  1. Bluehost.
  2. HostGator.
  3. InMotion.

---
Upgrading the specs or what is called **vertical scaling** is sometimes accepted if used on a database server being that you need more storage or a better CPU to process data ...

But what if, we still run into errors even after the upgrade?
many factors may come into play even with a maxed-out server, the CPU handling the threads of different requests simultaneously will cause it to prioritize some of them in favor of others causing unwanted/unexpected behaviors...

Imagine being in a restaurant that has 10 clients and a waiter that can't handle all incoming requests, so we replaced him with a four-armed waiter but that also didn't solve the problem being that, he takes a lot of trips to the kitchen and a client may order 4 plates that will occupy all 4 hands.

__Clearly__ the solution isn't surgically adding more hand to the waiter but in hiring another!
He'll take some load off the first and we'll be distributing the pressure on both.
That's what we're calling :

**horizontal scaling :**
adding more servers instead of maxing out the main server, which will consume a bit of the resources in each server making it easy to just add more, in a rush hour, and costing less than a **gigantic** server.

To apply horizontal scaling we need to deploy the code like we'ed normally do but on multiple servers *(let's say 10)*
Having the code duplicated on multiple places with __NGINX__ as a load balancer, it will distribute the incoming requests on our 10 servers

[leveraging 6 algorithms to decide which one to use :](https://www.nginx.com/faq/what-are-the-load-balancing-algorithms-supported/)
------
###### Weighted/Dynamic Round Robin.
###### Least Connections.
###### IP Hash.
###### Generic Hash.
###### Least Time.
###### Random.