teleren/devto huggingface dataset:
Life Expectancy Project
Tags: jupyter, python, datascience
Published: 2021-08-09 21:12:36

Here is another project from Codecademy. In this one we were given a csv file with information on the GDP and life expectancy(LEABY) for 5 countries and asked to find correlations. 

While this prompt didn't take too much skill to find the correlations, it was the first time I was just given a csv file and told to make a notebook with it while being given no specific steps to follow. 

While I am sure more visualizations could have been used, I wanted to keep it straight forward and to the point since I don't think much was needed to prove the point. However, I am not familiar with best practices for these kind of documents as this is my first one. I would love to hear your thoughts on what looked good and what could use improvement! 

https://github.com/TheHamhams/Life-Expectancy-GDP
#005 How to Convert JSON to NDJSON?
Tags: ndjson, bigquery, jq, json
Published: 2021-08-09 20:53:50

 > [NDJSON](http://ndjson.org/) is a convenient format for storing or streaming structured data that may be processed one record at a time.

###How to convert JSON to NDJSON using npm:

####Step 1. 

```
npm install -g json-to-ndjson
```

Node.js and npm are required!

####Step 2. 

```
cat file.json | json-to-ndjson
```

###How to convert JSON to NDJSON using [jq](https://stedolan.github.io/jq/) :

 **jq**  is a lightweight and flexible command-line JSON processor.

####Step 1.
Using this simple line of code, you can convert and save files in NDJSON format:
```
cat test.json | jq -c '.[]' > testNDJSON.json
```


###What we have got after?

We've just converted JSON array file.json (or a JSON object containing a JSON array), to NDJSON. When you use the NPM method, please note that the same file (file.json), we've just grind, become formatted as NDJSON! If you need to save the original JSON file, please create a backup before! 

###Why is NDJSON so important? 

*It works well with UNIX-style text processing tools and shell pipelines. 
*It's a great format for log files. It's also a flexible format for passing messages between cooperating processes. 
* It's also the download format that is used in **Google Big Query**.

### BigQuery loves NDJSON

These days I am working with BigQuery and before I create a data pipeline, test the process and I try to insert fat/big GB tables from PostgreSQL Data warehouse DB into BigQuery. 

OK, I completed the export jobs, exported a few tables from DB into JSON, saved them locally, and start to push them to the BigQuery.

Because they are larger than 4GB (GCP.BQ limit!) we must upload the data to Google Cloud Storage first, and as a second step after we created a dataset in the BigQuery service, we need to create the table from JSON file in our case. (This is one of the basic options to ingest data into and talk to BQ...) 

On this step I got the Error: **“Error while reading data, error message: Failed to parse JSON: Unexpected end of string; Unexpected end of string; Expected key”**

###WHY I GOT The error?

####BECAUSE: 
> BigQuery only accepts new-line delimited JSON, which means one complete JSON object per line

When you are importing data from JSON files into BigQuery, first you have to format the JSON file as NDJSON, otherwise, you will get the Error noticed above!

       
###Resources:
[ndjson.org](http://ndjson.org/)
[Why NDJSON](https://medium.com/@kandros/newline-delimited-json-is-awesome-8f6259ed4b4b)
[package - json-to-ndjson](https://www.npmjs.com/package/json-to-ndjson?activeTab=readme)
[BigQuery: Loading JSON data into a new table ](https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-json#loading_json_data_into_a_new_table)
Move Fast, Break Things & Win: How Facebook Builds Software
Published: 2021-08-09 20:48:01

Founder, professional poker player, podcaster and author, Jeff Meyerson, “broke his brain” to bring you the inside-story of Facebook.

Interviewing more than two dozen Facebook engineers, Jeff spent two and a half years writing his new book [Move Fast: How Facebook Builds Software](https://www.google.com/search?q=jeff+meyerson+Move+Fast%3A+How+Facebook+Builds+Software&rlz=1C1JZAP_enUS848US852&sxsrf=ALeKk00ivMbSB6KbO7TMGpD_-S1QgPQODg%3A1628534355776&ei=U3YRYenuLpuJwbkPz76NqAo&oq=jeff+meyerson+Move+Fast%3A+How+Facebook+Builds+Software&gs_lcp=Cgdnd3Mtd2l6EAM6BwgjELADECc6CQgAELADEAcQHjoHCAAQsAMQQzoICAAQgAQQsAM6BQgAEIAEOgoIABCABBCHAhAUOgUILhCABEoECEEYAVDvBliHC2DeFGgAcAB4AIABtwWIAZIJkgEHMC4zLjUtMZgBAKABAaABAsgBCsABAQ&sclient=gws-wiz&ved=0ahUKEwip5J6ey6TyAhWbRDABHU9fA6UQ4dUDCA4&uact=5). Written from Facebook’s own view of their software strategy and tactics, Move Fast explores the product strategy, cultural principles, and technologies that made Facebook the dominant social networking company.

Jeff joined Dev Interrupted to give us insight into his book and the secrets behind Facebook’s success, the founding story of the Software Daily podcast, the inside scoop on his new company SuperCompute, and how the skills and logic behind poker translate to software engineering.

#**Listen to the full episode**
{% spotify spotify:episode:2DlVeUI9A9NoXN1rg1zGzL %}

## Episode Highlights Include:

## 

* The intersection of poker and software engineering

* Founding story of the Software Daily podcast

* How to think about winning a market

* First steps to take when founding a company

* Product, culture and engineering — How do they relate?

* Is the Facebook the Goldman Sach’s of software?

* Why Facebook’s secret sauce is it’s onboarding process, not marketing

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nhfm87jqm0b10pu0y2gh.png)

**If you haven't already heard**, Dev Interrupted is partnering with Dzone to host **INTERACT:** An interactive, community-driven, digital conference on September 30th - by engineering leaders, for engineering leaders. 1 day, 10 speakers, 100s of engineers and engineering leaders, all free.

##[Register Now](https://devinterrupted.com/event/interact/)##

__________________________________________________________________


##Join the Dev Interrupted Server##

With over 1500 members, the Dev Interrupted Discord Community is the best place for Engineering Leaders to engage in daily conversation. No sales people allowed. [Join the community >>](https://discord.com/invite/devinterrupted)

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/a05i5erbpgoseb7nhlzd.png)

*Originally published at [https://devinterrupted.com](https://devinterrupted.com/podcast/move-fast-break-things-win-how-facebook-builds-software).*
How to replicate a PDF template into multiple pages
Tags: pdf, command, poppler, amazonkdp
Published: 2021-08-09 20:27:51

I've been using [poppler](https://poppler.freedesktop.org) for command line operations on PDFs, like split, merge, conversions, etc., as part of my workflow. My usual use-case is merging PDFs generated from SVGs, like described here in my article on [creating multipage PDF documents from SVGs](https://medium.com/@HEXcube/use-inkscape-to-create-multipage-pdf-documents-from-svgs-a1a11e6dce3d). Using commands for stuff like this can improve efficiency and productivity, whereas most GUI based methods would've made this process more time consuming and repetitive. So, when I found my significant other manually duplicating inner pages for her [Amazon KDP](https://kdp.amazon.com) notebook template, I set out to find a command to automate it.

Surprisingly, I couldn't find a built-in method on any CLI tool to do this directly. So, I used my Google-Fu and got this two liner working:

```sh
for i in {1..120}; do cp template.pdf template-$i.pdf; done
pdfunite template-*.pdf merged.pdf
```

The first command creates 120 copies of the template. Adjust the number as per your need. The second one then merges all these copies as pages into one PDF file. Make sure you have `poppler-utils` installed for `pdfunite` command to work. Windows users need to use a bash compatible shell like PowerShell or a Linux distro through WSL. 

## Credits & Sources
- Cover image from [directory pages](https://unsplash.com/photos/1yj3lRDWD6k) by <a href="https://unsplash.com/@igordresjan?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Igor Dresjan A.P.</a> on <a href="https://unsplash.com/?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Unsplash</a>
- [Linux / UNIX: Run Command a Number of Times In a Row](https://cyberciti.biz/faq/bsd-appleosx-linux-bash-shell-run-command-n-times/)
- [Automating the tedious with loops](https://librarycarpentry.org/lc-shell/04-loops/index.html)
- [How do I duplicate pages within PDF files automatically? - Solution with for loop](https://superuser.com/questions/600877/how-do-i-duplicate-pages-within-pdf-files-automatically/865818#865818)
- [How do I duplicate pages within PDF files automatically? - Solution with pdfunite](https://superuser.com/questions/600877/how-do-i-duplicate-pages-within-pdf-files-automatically/993634#993634)
New Beginnings
Tags: beginners, python, codenewbie
Published: 2021-08-09 20:21:24

As a dev, I find myself constantly thinking about the variety of languages that are out there and if there's one that I would "prefer" over the others. While I can see why that mindset could be a harmful one to a developers.....development, it's still a thought that occasionally crosses my mind. In cases like these it never hurts to feed the curiosity though. 

I recently started a Udemy Python Course. I decided to finally take the chance to see what it was all about. So far I've been able to successfully create a bandname generator. To achieve this, I had to become familiar with the print function, the input function and variable assignment within the language.

_Note: I've been working within an environment on [Replit.com](https://replit.com/) So for right now the playground is set up for me. As I learn how to run a script in console I will, then, be more detailed about all the proper steps._

Already bringing an understanding of strings from my time in JS and Ruby helped to get a quick understanding of using print(). Using it to print the greeting of the generator my first line of code was simply:

```
print("Welcome to the Band Name Generator.")
```
After, there are 2  questions that gets asked about the name of your hometown and the name of your pet. Following both questions, on a new line you see the program continuing to run awaiting for the user to type the response and press enter. After inputting both responses the user is then met with their possible band name. 

```
city = input("What's name of the city you grew up in?\n")

pet = input("What's your pet's name?\n")

print("Your band name could be " + street + " " + pet)
```

The final print statement, without variable assignment, could definitely be a very wordy line of code. Instead, we assign the results of the input function(what the user inputs when prompted) to the variables city and pet. Finally we use string concatenation within the second print function to tie the final line of the generator together. 

_Note:With concatenation you should be mindful to think of spacing within the sentence._

```
Your band name could be Charleston Folly
```
Vim 101: Basics of the Vim Text Editor
Published: 2021-08-09 20:14:44

At the Frederick Open Source meetup this month, I talked about the Vim text editor. I covered a lot of the fundamentals, including a mental framework to help demystify why Vim is challenging to so many people. I also covered some tips that people can use to work toward Vim mastery.

The recording from the talk is available on YouTube. [Check it out!](https://www.youtube.com/embed/zxdQJlB65lA)

{% youtube zxdQJlB65lA %}
Difference Between Aggregation and Composition
Tags: java, aggregation, composition, oop
Published: 2021-08-09 20:09:31

Aggregation and Composition are basically the concepts about how to manage objects. 

## Aggregation
Aggregation uses *loose coupling*, which means that it does not hold the actual object in the class. Instead, the object is being passed via getter or setter function. It gives the benefit that when the object of the parent class dies then the child class object remains intact.

```java
class House {
 private Table table;
 
 public void setTable(Table table){
  this.table = table
 }
 
 public Table getTable(){
  return this.table;
 }
}

public class main(String[] args){

 Table table = new Table();
 
 // this house' table is set externally and will not be deleted 
 // upon deleting this House object
 House house = new House();
 house.setTable(table);

}
```
## Composition 
Composition uses *tight coupling*, which means that it holds the object. An instance variable is created and the object is stored in it. Whenever the object of the parent class removed/deleted then the child class object will also be deleted.

```java

class Human {

 // This heart will be deleted and nowhere to access once its 
 // human gets deleted
 private Heart heart = new Heart();

}

public class main(String[] args){
 
 // this human's heart is set internally and will be deleted 
 // upon deleting this Human object
 Human human = new Human();

}
```
Remind Yourself Of The Original Strategy Every Day
Published: 2021-08-09 19:35:53

I have a fragile memory, and the internet is too secure for me. I get distracted more or less every hour with some life-changing new strategy that may improve my SAAS business and gets me hundreds of clients an hour with some new ‘growth hack’s. The original plan based on a well thought out strategy is just forgotten.

I find it super useful to wake up and read a simple document that outlines the strategy and why.

For example, one of our marketing strategies is inbound marketing. So amongst others, I remind myself of the logic and reasons behind this:

“We will rely on content creation and inbound marketing to first create value and then earn visits to our website. Content is the king as the value it creates is permanent and grows with time. It is a real asset. We are in no hurry to make a quick buck. Plus, we want to build goodwill in the industry, not just earn a customer.”

It reminds me and allows me not to get tempted by the latest growth hack. It allows me to leave money on the table by not getting busy with over-optimization and have the right perspective not to feel anxious about it or that I am missing out.

We do that with our product strategy, how we handle the team, how we feel support. It grounds us, calms us down to the fundamentals every single day. I firmly believe it is one of the main reasons we have been able to make a success out of our SAAS service, Proxies API.

The author is the founder of Proxies API, a proxy rotation api service.
Designing a Rate Limiter
Tags: architecture, technology, systemdesignintervie, softwareengineering
Published: 2021-08-10 04:48:13

---
title: Designing a Rate Limiter
published: true
date: 2021-08-09 19:35:30 UTC
tags: softwarearchitecture,technology,systemdesignintervie,softwareengineering
canonical_url: https://towardsdatascience.com/designing-a-rate-limiter-6351bd8762c6
---

[![](https://cdn-images-1.medium.com/max/2600/0*g9mqvv0jpR7XbbB2)](https://towardsdatascience.com/designing-a-rate-limiter-6351bd8762c6?source=rss-577e2119c7f2------2)

How do you design a rate limiter service?

Rate limiting is a simple concept to grab, but it’s vital for large-scale systems. It is used for security purposes as well as performance improvement. For example, a service can serve a limited number of requests per second. However, if a service is receiving a huge number of requests at a time, it might be too much for the server to handle. In that case, even the server might be shut down. To handle this type of problem, we would need some kind of limiting mechanism that would allow only a certain number of requests to the service.

###Why do Need Rate Limiting?
Rate limiting is needed to protect a system from being brought down by hackers. To know the importance of rate-limiting, we have to know about the DOS attack.
The denial of service attack, which is mostly known as a DOS attack, is when hackers try to flood a system with many requests within a short period of time to shut the system down. Because a server has a limit of how many requests it can serve within a certain period of time. So, the system can’t handle the floods of requests properly; it can not handle them properly.
Rate limiting can prevent that from happening as it protects the system from being flooded with intentional unnecessary requests. After a threshold value, the system returns an error.
###Sliding Window Algorithm:
If we keep track of each request per user in a time frame, we may store the timestamp of each request in a Sorted Set in our ‘value’ field of hash-table. But, it would take a lot of memory. So, we can use a sliding window with the counters. Now, consider this if we keep track of request counts for each user using multiple fixed time windows.
For example, if we have an hourly window, when we receive a new request to calculate the limit, we can count requests each minute and calculate the sum of all counters in the past hour. This would reduce the memory we need for the set.
Suppose we rate-limit at 500 requests per hour with an additional limit of 10 requests per minute. This means that the client has exceeded the rate limit when the sum of the counters in the past hour exceeds the limit request(500).
For a smaller time frame, the client can’t send more than ten requests per minute. This would be a hybrid and reasonable consideration as we may think that none of the real users would send such frequent requests. Even if they do, they will get success with retries since their limits get reset every minute.
[Continue reading on Towards Data Science »](https://towardsdatascience.com/designing-a-rate-limiter-6351bd8762c6?source=rss-577e2119c7f2------2)
Validando associações com Ecto: put_assoc vs cast_assoc
Tags: elixir, ecto
Published: 2021-08-09 19:12:42

Recentemente fazendo algumas refatorações de código, identifiquei um problema onde a validação de alguns changesets não estavam funcionando, e olhando a implementação tudo estava aparentemente correto:

```elixir
def create_user(params) do
  User
  |> User.changeset(params)
  |> put_assoc(:personal_info, params.personal_info)
  |> Repo.create()
end
```

Debugando o código, vi que o problema era que ao executar o `create_user`, o changeset do `personal_info` não era executado. Então fui ver na documentação, e vi que a `put_assoc` exige que os parâmetros tenham sido validados pelo `changeset` antes da chamada da função, então o que eu precisava fazer era o seguinte:

```elixir
def create_user(params) do
  User
  |> User.changeset(params)
  |> put_assoc(:personal_info, build_personal_info(params))
  |> Repo.create()
end

defp build_personal_info(%{
  personal_info: personal_info
}) do
 PersonalInfo
 |> PersonalInfo.changeset(personal_info)
end
```

E dessa forma a validação passou a ser feita da maneira esperada. Porém, eu adicionei mais uma função privada no meu contexto. Então, eu fui atrás de uma forma melhor de resolver esse problema. A solução foi utilizar a `cast_assoc` diretamente no `changeset` do `User`, pois ela faz a relação dos dados e trata as informações com o `changeset`. E como os parâmetros já possuíam a chave `personal_info`, eu não precisei informar qual parâmetro
deveria ser validado, pois a função já faz isso:

```elixir
  defmodule MyApp.Accounts.User do
    ...

    schema "users" do
      ...
      # Atente a necessidade da relação estar declarada no schema
      has_one(:personal_info, PersonalInfo, on_delete: delete_all)
    end
    
    def changeset(user, attrs)
      user
      |> cast(attrs, @optional_fields ++ @required_fields)
      |> validate_required(@required_fields)
      |> cast_assoc(:personal_info, required: true)
  end
```

Dessa forma, eu não precisei fazer nenhuma chamada no Contexto, nem criar uma nova função para validar os dados de `PersonalInfo` ao salvar dados de uma nova pessoa usuária.
Girls In Tech 
Published: 2021-08-09 19:11:47

 **“We each have a unique path to walk in this life, and there is a reason that yours is unfolding the way that it is. Embrace your journey and look for the lessons. Believe in divine timing and know that what’s for you will not pass you.”**



Why do people think only boys can do code? Why can't girls do it?
This was the wrong thinking of the people they had. Why do people discriminate between boys and girls? If boys can do it, then why can't girls do that? If girls have an interest in Computer Science, then why did people say don't pursue your career in Computer Science? 
This happened because of less knowledge in the Computer Science field, they don't know what they will do in this field.

**Computer Science is a field where you can do lots of technology work, which will increase your productivity. The study of algorithmic processes, computational machines, and computation itself. As a discipline, computer science spans a range of topics from theoretical studies of algorithms, computation, and information to the practical issues of implementing computational systems in hardware and software.** 


People make computer science look like a man's game, but it isn't. Those girls have an interest in coding, but they can't do it. With the persistence of male domination in the tech industry, it’s easy to forget how instrumental and spearheading women have been in its development since its origins around 300 years ago. Despite the constant barrage of obstacles, female scientists and mathematicians have persisted in order to have their work recognized and widely used, and as a result, many of the pioneers in coding, but they can't do it.
Jump to running tmux sessions with fzf
Tags: cli, linux, tmux
Published: 2021-08-09 19:05:20

---
title: Jump to running tmux sessions with fzf
tags: ['cli', 'linux', 'tmux',]
series: tmux
canonical_url: https://waylonwalker.com/tmux-fzf-session-jump/
published: true
---

Quickly getting between tmux splits is critical skill for productivity.  You
can get by with `next` or `prev` session for awhile, but if you have more than
about three session you need something a bit more targeted.


## Full Screen selector

I have used this fzf one keybinding for quite awhile,  honestly I did not make
it up, and cannot remember where it came from. It will open up a session picker
in a new full screen window.

``` bash
bind C-j new-window -n "session-switcher" "tmux list-sessions | sed -E 's/:.*$//' | grep -v \"^$(tmux display-message -p '#S')\$\" | fzf --reverse | xargs tmux switch-client -t"
```

## Popup selector

Like with many of my keybindings I have swapped this one out for a popup
version.  It just feels so smooth.

``` bash
bind C-j display-popup -E "tmux list-sessions | sed -E 's/:.*$//' | grep -v \"^$(tmux display-message -p '#S')\$\" | fzf --reverse | xargs tmux switch-client -t"
```

Be sure to check out the full YouTube playlist and subscribe if you like it.

[![tmux playlist on youtub](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/v4cgo76y53s1hwykajo1.png)](https://www.youtube.com/playlist?list=PLTRNG6WIHETB4reAxbWza3CZeP9KL6Bkr)

{% post https://dev.to/waylonwalker/how-i-navigate-tmux-in-2021-2ina %}

> Also check out this long form post for more about how I use tmux.
Notes on Operating System
Published: 2021-08-09 18:47:52

### System Software
* Provide a platform to other software.
* Manage resources and helps to run hardware and application software.
* Runs when system starts and runs till end.
* Essential for proper functioning of a system.
For example :- OS, language processor and device drivers.

### Application Software
* Perform a group of coordinated functions, tasks and activities.
* Perform a specific task according to their type.
* Runs when user required.
* Not extremely important for functioning of system.
For example :- Word Processor, Spreadsheet, Web Browser etc.

To be continue...