teleren/devto huggingface dataset:
Explaining Data Engineering to a 5 Year-Old (data is mud)
Tags: datascience, beginners, database
Published: 2022-02-12 16:44:26

You are at the playground, and it starts raining. What happens to the soil? It gets muddy - and then you can try to play with it and make shapes out of it. Maybe get a bit creative and colors it or add engravement? 

What does that have to do with data engineering? Well, data in this story is mud, and **trust me data can get muddy.** 

> A data engineer's job is to collect the mud, see if it can be used to create clay. Remove any leaves or rocks from it. Then figure out where to put it in a place so that it stays in a good condition. Finally, they do their best to ensure that the mud is as clean as possible, and is stored in a safe and comfortable environment.  

In data terms, data engineer's responsibilities are but are not limited to:

**- Data Collection** (collecting mud)
They are responsible for and sometimes advise on the best tools used for data collection. Support in the data collection process. Ensuring that the collected data is as clean and error-free as possible.

**- Data Storage** (storing the mud for easy access)
They would be responsible for storing the data. It is not an easy task, they need to ensure that data is stored in a way that is most suitable for the data type. For example, videos and images are stored differently than text. Also, they need to ensure that data retrieval is as optimal as possible by data analysts and data visualist. They also need to ensure that data is stored securely and protected from any breach attempt. 



**- Data Clean up** (removing leaves and rocks from the mud)
They are sometimes responsible for cleaning data entry mistakes, outliers, discrepancies in the data to some extent. 

**- Data Analysis** (figuring out the mud type and what we can make of it)
Though this is the job of a data analyst. Data engineers do not always get the joy of digging deep into the data and understanding its hidden messages. But they do take on primal data analysis. 


On a closing note, data engineers may not be at the frontline, but they pave the road for data visualization and analysis. They are the foundation that data science is built on. They are our data avengers.


![Data Security](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ia4r0ngjolkee79e92xx.jpg)


How would you describe a data engineer?
Webauthn - Introduction to passwordless authentication
Tags: auth, webauthn, passwordless
Published: 2022-02-12 16:37:12

> This is intended to be the first part of an article serie. If you like it, please add a reaction to encourage me to write the subsequent more in-depth articles.

For those who don't know, there is a new protocol under way: "webauthn"

You can try it out here: https://webauthn.io/

Basically, it leverages device authentication (using biometrics like touch or face recognition) to generate cryptographic private/public keys and sign in using those.

The registration flow looks like this:

![registration](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4q8p3kzzrajo4n7prqzx.png)

And once your device is registered, it can use its private key to sign in:

![login](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lic9uyxq9wxtwww551ez.png)

Sounds wonderful right? Well, I agree! A world without passwords is a better world. 

And since the "principle" looks simple, its usage and APIs should be simple too, right? Well, not quite... The protocol itself is still fairly young and when you go to the official specifications also known as "RFC", you will face an impressive wall of text.

- The RFC Currently: https://w3c.github.io/webauthn
- The Repository: https://github.com/w3c/webauthn

Most of it being technical, verbose and IMHO very challenging. Currently, printing the RFC on plain paper would produce 165 pages! Impressive right?
 
![rfc joke](https://i.imgflip.com/1c6f6g.jpg)

And this complexity propagates everywhere, even to the interface itself. You do not deal with normal JSON objects that you can simply send to the server, but with native byte array, exotic CBOR (Compact Binary Object Notations) encoding, a multitude of different attestation types, unclear content...

This series is about understanding this protocol better, evaluating a few libraries, implementing passwordless on one's own server, strategies for "one account / multiple devices", what is undergoing change, and so on. The next topic will be "Webauthn - the client side". Stay tuned.
Deadlock in mysql
Tags: mysql, database, sql, postgres
Published: 2022-02-12 16:36:44

A deadlock in mysql happens when two or more transactions are mutually holding and requesting locks on the same resources, creating a cycle of dependencies. Deadlocks occur when transactions try to lock resources in a different order. For example, consider these two transactions running against the orders table:
Transaction #1
START TRANSACTION;
UPDATE orders SET price = 50 WHERE id = 2;
UPDATE orders SET price = 60 WHERE id = 6;
COMMIT;

Transaction #2
START TRANSACTION;
UPDATE orders SET price = 60 WHERE id = 6;
UPDATE orders SET price = 50 WHERE id = 2;
COMMIT;

If you are unlucky, each transaction will execute its first query and update a row of data, locking it in the process. Each transaction will attempt to update its second row, only to find that it is already locked. The two transactions will wait forever for each other to complete, unless something intervenes to break the deadlock.

to solve this problem, database systems implement various forms of deadlock detection and timeouts. the InnoDB storage engine will notice circular dependencies and return an error instantly. This can be a good thing otherwise, deadlocks would manifest themselves as very slow queries. others will give up after the query exceeds a lock wait timeout, which is not always good. The way InnoDB currently handles deadlocks is to rollback the transaction that has the fewest exclusive row locks.

Lock behavior and order are storage engine specific, so some storage engines might deadlock on a certain sequence of statements even though others won’t.
Deadlocks have a dual nature:
- some are unavoidable because of true data conflicts.
- some are caused by how a storage engine works.

Deadlocks cannot be broken without rollingback one of the transactions, either partially or wholly. They are a fact of life in transactional systems, and your applications should be designed to handle them. Many applications can simply retry their transactions from the beginning.
Angular Material UI/Theme Builder
Tags: angular
Published: 2022-02-12 16:29:28

Over the past year, we have been developing a  drag-and-drop Angular Material UI/Theme Builder. The target audience is angular developers who already have experience with Google's Angular and Angular Material Components.

The Angular Material UI/Theme builder is a What-You-See-Is-What-You-Get (WYSIWYG) drag-and-drop builder that allows angular developers to build reactive forms and views by dragging and dropping Google's Angular Material Components onto an Angular Flex layout. The properties, styles, and events of each component can be configured through panels and the builder generates all the code.

The code that is being generated can be viewed through three tabs: (i) Markup tab for the HTML, (ii) Script tab for TypeScript and (iii) Styles tab for the CSS.

The UI Builder also incorporates Angular Material Theme Builder that can be used to build a theme and apply the theme to the views you have built using the UI Builder.

You can also bind the views/forms to static data or ReST-API based data sources.

The Table UI Component in the Angular Material library is very basic and not sufficient for most applications. Therefore, in addition to the Angular Material Components, the builder also integrates ag-grid (https://www.ag-grid.com) for Grid Control. 

The goal of this UI/Theme Builder is to reduce the time to develop a view by over 50%. Again, this UI/Theme Builder is targeted towards developers who already have experience with Angular and Angular Material.

I am looking for the community feedback on whether or not you would use such a UI/Theme Builder. Secondly, would you be interested in trying the UI/Theme Builder and providing us feedback?
Add nestjs nx plugin, create server app and core lib
Tags: kaufmanbot, nx, nestjs
Published: 2022-02-12 16:10:33

## Install nestjs plugin to nx monorepo

> npm install -D @nrwl/nest

```sh
endy@endy-virtual-machine:~/Projects/current/kaufman-bot$ npm install -D @nrwl/nest

added 158 packages, and audited 623 packages in 18s

78 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities
```

## Add nx script to package.json

```
"scripts": {
    "nx": "nx",
    "start": "nx serve",
    "build": "nx build",
    "test": "nx test"
},
```

## Create server

> npm run -- nx g @nrwl/nest:app server

```sh
endy@endy-virtual-machine:~/Projects/current/kaufman-bot$ npm run -- nx g @nrwl/nest:app server

> kaufman-bot@0.0.0 nx
> nx "g" "@nrwl/nest:app" "server"

UPDATE workspace.json
UPDATE nx.json
CREATE jest.config.js
CREATE jest.preset.js
UPDATE package.json
UPDATE .vscode/extensions.json
CREATE apps/server/src/app/.gitkeep
CREATE apps/server/src/assets/.gitkeep
CREATE apps/server/src/environments/environment.prod.ts
CREATE apps/server/src/environments/environment.ts
CREATE apps/server/src/main.ts
CREATE apps/server/tsconfig.app.json
CREATE apps/server/tsconfig.json
CREATE apps/server/project.json
CREATE .eslintrc.json
CREATE apps/server/.eslintrc.json
CREATE apps/server/jest.config.js
CREATE apps/server/tsconfig.spec.json
CREATE apps/server/src/app/app.controller.spec.ts
CREATE apps/server/src/app/app.controller.ts
CREATE apps/server/src/app/app.module.ts
CREATE apps/server/src/app/app.service.spec.ts
CREATE apps/server/src/app/app.service.ts

added 230 packages, removed 5 packages, changed 3 packages, and audited 848 packages in 43s

105 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

up to date, audited 848 packages in 3s

105 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities
```

## Create core library

> npm run -- nx g @nrwl/nest:lib core/server

```sh
endy@endy-virtual-machine:~/Projects/current/kaufman-bot$ npm run -- nx g @nrwl/nest:lib core/server

> kaufman-bot@0.0.0 nx
> nx "g" "@nrwl/nest:lib" "core/server"

CREATE libs/core/server/README.md
CREATE libs/core/server/.babelrc
CREATE libs/core/server/src/index.ts
CREATE libs/core/server/tsconfig.json
CREATE libs/core/server/tsconfig.lib.json
UPDATE tsconfig.base.json
CREATE libs/core/server/project.json
UPDATE workspace.json
CREATE libs/core/server/.eslintrc.json
CREATE libs/core/server/jest.config.js
CREATE libs/core/server/tsconfig.spec.json
CREATE libs/core/server/src/lib/core-server.module.ts
```
![npm run -- nx g @nrwl/nest:app server](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/eopjvbg3dzczgubxq60k.png)
Hello World_
Published: 2022-02-12 15:59:24

I discovered developer blogging more than a year ago and it intrigued me but for whatever reason, I did not start one. It was a blunder on my part but as they say, better late than never.
My goal with this blog is to post once a day. No matter how busy I am. No matter how small the blog post will be. I just need to show up and post something.

With all that said, ***My blogging journey will now start in 3... 2... 1..!***
Hello, World!
Tags: firstpost, introduction, hello, world
Published: 2022-02-12 15:55:38

Well, this is the first step to doing anything really if you are a programmer.

`Hello, World!`

I'm TheAwesomeGuy47, an avid programmer and also the best darn MLG Kermit to exist.

> TheAwesomeGuy47 (2 seconds later)
> WAIT WHAT DO YOU MEAN I'M THE ONLY ONE? GONZO, YOU'RE FIRED.

alright now Gonzo has been fired, let's talk about what I know (at the time of writing) and what I want to learn.

[This link has the information](https://github.com/TheAwesomeGuy47)

Well I guess that's it. Goodbye and once again,

`Hello, World!`
Discoveries in Ruby(and Rails): List local/global_variables in the current scope, and the (_) in irb sessions. 
Tags: ruby
Published: 2022-02-12 15:50:23

Ruby provides a mechanism for tapping into the program at runtime to determine the list of defined local and global variables. Ruby provides two methods to get all declared variables in a local or global scope.

## Kernel#local_variables

The `local_variables` is available globally in your ruby programmes. 

```ruby
x = 5
puts local_variables
# => [:x]
```

the `local_variables` method will get the variables in the current local scope. So, for something like

```ruby
y = 6
def do_something
  x = 5
  puts local_variables
end

do_something
local_variables
```

The first line printed to the console will be the local variables inside `do_something`. The second print will be the local variables in the current scope, namely `y`.

## Kernel#global_variables

In Ruby, a global variables is a variable that starts with a dollar sign `$`. Ruby has a set of built-in globals available for us already.

```ruby
puts global_variables
```

You can add to the list of global variables by creating your own global variables, tho you might not want to do that.

```ruby
$my_global_variable = "Custom global"
global_variables.find { |glob| glob == :$my_global_variable }
 # => :$my_global_variable 
```
**Notice:** All variables are stored as a Symbol.

### Bonus content, the `_` local variable in irb sessions.

When you launch an irb session and log the `local_variables` you get an `_` in the list. The `_` provides a way to access the last return value in case it's lost forever.

You can access the return value of the previous line by using `_` in your irb sessions.

```ruby
x = 5
y = _

puts x 
puts y
```

Running this through irb you get `5` logged two times. That is because `_` stores the last return value, which is the assignment of 5 to the variable x.


### Resources

- [Kernel#local_variables](https://apidock.com/ruby/v2_6_3/Kernel/local_variables)
- [Kernel#global_variables](https://apidock.com/ruby/v2_6_3/Kernel/global_variables)
Setup python virtual environment in linux
Published: 2022-02-12 15:41:41


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lnqyo75nkdmqagc4vxbc.jpeg)
Virtual environment is a very important tool to isolate each project modules and dependencies from other projects in our system.
Let's see how to setup python virtual environment in a linux based system.

1) First let's install virtualenv module. Open the terminal and type the below command.
`$ pip install virtualenv`

2) Then let's check if our virtualenv installation is successful.
`$ virtualenv --version`
And it should show the version and location of the virtualenv installation.
`virtualenv 20.13.0 from /usr/local/lib/python3.6/dist-packages/virtualenv/__init__.py
`
3) Now it's time to create a virtual environment.
Move to the folder where you want to setup your virtual environment and type the following command.
`$ virtualenv my_venv`
Here 'my_venv' is the name of the virtual environment. You can give any name as you like.

4) After creating a 'my_venv' named virtual environment in our project folder, now let's activate our virtual environment. Enter the following command to activate the virtual environment.
`$ source my_venv/bin/activate`

After activation the name of your virtual environment will appear in the left side of your terminal like this
`(my_venv)`

Now you can install dependencies required for our projects. And now your virtual environment will isolate all the dependencies that we have installed for this project.
I decided to try to keep a twitter history of rewriting one of the projects, I'm starting ...
Tags: kaufmanbot, nx
Published: 2022-02-12 15:30:45

I decided to try to keep a twitter history of rewriting one of the projects, I'm starting...

Creating an empty NX project

https://github.com/EndyKaufman/kaufman-bot/blob/feature/73/STEPS.md

> npx create-nx-workspace

![npx create-nx-workspace](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/t3q0182aqcfigzai7h5u.png)
Selenium Certification Course
Tags: selenium, seleniumonlinetraining, seleniumcourse
Published: 2022-02-12 15:01:25

As the world moves quickly from manual testing to automation testing, the demand for automation testers has increased in the past few years. 
Do you want to be an Automation tester? Selenium is the #1 tool you must master as it is flexible, in high demand and completely free. In this [Selenium Certification Course](https://www.h2kinfosys.com/courses/selenium-webdriver-junit-training-course/), you will learn the processes involved in writing test cases that work.
Improving the user interface of my calculator app, kotlin Android studio
Tags: kotlin, app, android, ui
Published: 2022-02-12 14:42:22


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/47wtepjxzwmhs9rsjkn3.jpg)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rqx5y1ci542kl9p8xia3.jpg)
Why making Umbraco headless makes sense
Tags: umbraco, graphql, dotnet
Published: 2022-02-12 14:32:22

When choosing a CMS system, it's common to look at the features a given system provides. You might consider the editor experience or how you could integrate clients' existing content providers, for example, video platforms, user databases, or CRM systems.

But most importantly, you consider what the development process would look like for you as a developer. Are you using the technologies you want to use, and does it give you the extensibility you need?

When looking at the [Umbraco CMS (v9+)](https://umbraco.com/) you find many ways to integrate and utilize existing systems in the CMS journey and an overall smooth editor experience. You get access to newer technologies like .Net 5+ and, thereby, everything that comes with the evolving .Net 5+ framework.

But one element that might discourage many developers from choosing this CMS is the absence of headless integration. Because although you get many features and a great editor experience, the frontend limitations can be challenging for non-Dotnet frontend teams. It also excludes a whole swath of modern frontend frameworks from being used without developing a custom headless integration or working within the Razor templating language.

Because of this, the [Umbraco HQ](https://umbraco.com/) created the [Umbraco Heartcore](https://umbraco.com/products/umbraco-heartcore/) project that builds upon the existing Umbraco CMS by adding a headless integration in GraphQL. The only problem with this solution is the pricing. Because Umbraco CMS is open-source and free to use, you might see this product solution as a barrier to entry. It also makes it impossible to use your infrastructure to manage your CMS as they require the usage of Umbraco Cloud.

So although the CMS has a headless version, this doesn't give the same flexibility that you can get from the open-source project. Therefore, multiple projects have been started by the Umbraco community to make the CMS headless with either a simple endpoint to get content data or a more complex GraphQL integration for a more custom output. Unfortunately, many of these projects lack production viability and or documentation, making them inaccessible to the greater Umbraco community.

I've therefore taken the assignment upon myself to create a GraphQL integration to open the headless CMS possibilities for Umbraco. The project is still in the early stages but already offers queries for content and their properties, and it's built on the [Hot Chocolate](https://chillicream.com/docs/hotchocolate) framework making it easy to extend and secure.

{% github https://github.com/nikcio/Nikcio.UHeadless %}
Spring Boot banner
Tags: tutorial, springboot, java, programming
Published: 2022-02-03 07:49:15

![springboot](https://cdn-images-1.medium.com/max/2000/0*nPHVux9W1NhTpySv.png)

Usually when you start a Spring Boot application you have something like this in your console output :

      .   ____          _            __ _ _
     /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
    ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
     \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
      '  |____| .__|_| |_|_| |_\__, | / / / /
     =========|_|==============|___/=/_/_/_/
     :: Spring Boot ::                (v2.5.5)

But did you know that you can disable it, or even create a custom one ?
No ? Yes, and it’s simple, so let’s go.

## Turning banner off

If you don’t want the banner to show during the application launch you have two options :

* Setting the property to off in your *application.properties* file

    spring.main.banner-mode=off

* In your main class, change the line to run the application from this

    SpringApplication.run(BannerApplication.class, args);

to this

    new SpringApplicationBuilder(BannerApplication.class)
            .bannerMode(Banner.Mode.*OFF*)
            .run(args);

## Having a custom banner

By default Spring Boot will search a file named banner.txt in your ressources folder to use it, but your banner can also be an image (that will be transformed in ASCII), so if you want to use an image you should add a property in your *application.properties* file

    spring.banner.image.location=**classpath:banner.png**

here I used a image of Stich as a banner, and had this output

![stich](https://cdn-images-1.medium.com/max/2528/1*i8hz-NZtXJqZSV--DLZQDg.png)

Not very beautiful nor readable, so let’s see what I have if i use a *banner.txt* instead of an image.

So, I created in my ressources folder, a *banner.txt* file, in witch i wrote “Hello the World !!” and then started my project

![](https://cdn-images-1.medium.com/max/2000/1*Ul5OUqtIT82TilDkRIZ4Lg.png)

Simple and efficace but, we can do better, some site provide you a banner generator that let you choose the text and the font, I have faith in your use of Google to find them, but as an example you can have something like that :

![](https://cdn-images-1.medium.com/max/2628/1*F7npg3HRZBDz0txCNa0sOw.png)

You can use place holder such as thos to add some informations in your banner.
{% gist https://gist.github.com/ErwanLT/2c4b7da4eadb0d54342d1023b1a899c4.js %}
In addition, the text can be coloured with variables such as ${AnsiColor.NAME} or ${AnsiBackground.NAME}.