teleren/devto huggingface dataset:
How does DAG work in Spark?
Tags: spark, computerscience, mathematics, datascience
Published: 2022-08-21 18:43:00

Directed Acyclic Graph (DAG) is a finite directed graph with no directed cycles. There are finitely many vertices and edges, where each edge is directed from one vertex to another. DAG contains a finite set of vertices and edges in sequence. Every edge in DAG is directed from top to bottom in the sequence.

This is why it's great for generating multistage scheduling layers that implement stage-based scheduling. As the number of layers and depth can be more than two but finite, it's better and optimized than the older Map and Reduce programs, which run in two stages only: Map and Reduce.

**Working of DAG Scheduler**
The interpreter is the first layer, using a Scala interpreter, Spark interprets the code with some modifications. Spark creates an operator graph when you enter your code in the Spark console. When we call an Action on Spark RDD at a high level, Spark submits the operator graph to the DAG scheduler.

Divide the operators into stages of the task in the DAG scheduler. A stage contains tasks based on the partition of the input data. The DAG scheduler pipelines operators together. For example, map operators schedule in a single stage. The stages pass on to the task scheduler. It launches tasks through cluster manager. The dependencies of stages are unknown to the task scheduler.

The workers execute the task on the slave. The following diagram briefly describes the steps of how DAG works in the Spark job execution:


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lbx5qa0i177hjtkavbbg.png)

**Driver program:** The Apache Spark engine calls the main program of an application and creates Spark Context. A Spark Context consists of all the basic functionalities. RDD is created in the Spark Context.

**Spark Driver:** It contains various other components, such as DAG scheduler, task scheduler, backend scheduler, and block manager, which are responsible for translating the user-written code into jobs that are actually executed on the cluster.

**Cluster manager:** Cluster manager does the resource allocating work.

Hope this was helpful.
Active Record Associations
Published: 2022-08-21 18:41:00

**What are Active Record Associations?**

Association is the relationship between models and it's used to make it easier on coders like you. Let's dive into the different types of associations with a example for each.

## The 6 Types of Associations

1.**belongs_to**
A belongs_to association creates a connection with another model. The associations has to be used in a singular term because Rails will automatically infer the class name from the association name. In this example, each review can be assigned to exactly one user. It will look for a user_id on the reviews table.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cga94aksdinfoj26razh.png)

2.**has_one**
A has_one association indicates that one other model has a reference to this model. For example, each supplier only has one account. The account will be associate with supplier_id.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qwamchsrz0w4mzcfskql.png)

3.**has_many**
A has_many association is a one-to-many connection with another model. The associations has to be used in a plural term. For example, a user can have many reviews in this example.
 
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/h5mks79tk57grkvy1pqd.png)


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/63e0drns1msspa31yrvb.jpeg)

4.**has_many :through**
A has_many :through association has a many-to-many connection with another model. The models are connected _through_ another model. For example, a game **has many** users, **through** reviews and a user **has many** games, **through** reviews. A review **belongs to** a game and a user. 


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/51sko6bt4gof4ndfa8et.png)

The tables will look something like this.
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/o99ds25ps79lo7y6omz3.png)

5.**has_one :through**
A has_one :through association sets a one-to-one connection with another model but can proceed through a third model. For example, if a department **has one** manager, and each manager is associated with one manager history, then it will look like this: 


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/57hcla8t8x7648lk65e8.png)
 
6.**has_and_belongs_to_many**
A has_and_belongs_to_many association is similar to the has_many :through association, but there is no third model. For example, there was a review model, but now there is only a game and user model.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1wj04rk3zhm1k06xtmk2.png)

There is also a polymorphic association that is more advanced and to learn more about that I would recommend visiting [this](https://guides.rubyonrails.org/association_basics.html). I hope this was helpful and good luck coding!

Sources:
-https://guides.rubyonrails.org/association_basics.html
-https://www.youtube.com/watch?v=5mhuNSkV_vQ&t=409s
Time Management as a student and a freelance programmer
Published: 2022-08-21 18:38:37

Hi, my name is Saifullah Usmani. I am a  freelance Programmer. I have been programming for almost 6 years now. And I make websites (SaaS commonly) for my clients.


My mid-session exams just started. I have 3 SAAS projects and a load of stuff to study. I got to save grades and keep my reputation in front of my clients at the same time. Both of the things are equally important to me. I have to do the basics things in life too. I got to sleep well, I got to do exercise and spend some time with family and most importantly with myself too to stabilize all the chaos in the mind.
Suddenly I started doing so many things and I felt my mind hurting. I was trying to meet every expectations. I will study hard and become a topper of my class. I will make the SAAS my clients have never seen and they will say WOW instantly. I will do exercise like never before, I will transform into my dream self. I will study productivity, money making, personal development books and a lot more.
But tbh study alone takes all of my time because I am not very good at it so I got to spend more time to get better at it. The subjects are different and new to me. So I cannot do all of the other stuff, this causes tension.
I have to manage my time. But what is time management exactly? Do I have to do all of it in 24 hours? Can I do it in 24 hours? Theoretically, Yes! practically, No?
So I have concluded that somethings will get sacrificed. I have to let go of perfection and skip some things to make time. Time management doesn't work with too many things. It breaks. It becomes Time Allocation. Allocate time to not different parts of days but rather allocate time to different days of week.
Exercise 2 times a week rather than 5 times or everyday. Go out less, work on important projects more. Take more time from clients, Communicate. Dedicated time of day to study. Study throughout the day on different intervals. You cannot study all at once. Expand it over the day. Consume whole day rather than part of day. Keep working hard as this time will pass away too.

I think I have an exam tomorrow so I better study😁
Love you guys, see ya all soon! Bye❤




[![DigitalOcean Referral Badge](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/00rcc3tqicv68383vo0r.png)](https://www.digitalocean.com/?refcode=9d6f3f71c1b4&utm_campaign=Referral_Invite&utm_medium=Referral_Program&utm_source=badge)
Affiliate Link☝




##Follow me:
Hire me: https://www.fiverr.com/saifullah02 (For web development Purposes)
My New Blog: [saifullahusmani.com](https://saifullahusmani.blogspot.com/)
*Follow me here:* https://dev.to/saifullahusmani
*YouTube:* http://bit.ly/saifullahusmani (Programming Tutorials Coming Soon).
*Discord:* https://discord.gg/AKyvy8Y
*Facebook*: [https://www.facebook.com/SaifullahUsmani693](https://www.facebook.com/SaifullahUsmani693)
Wheel Of Fortune
Tags: redishackathon
Published: 2022-08-21 18:07:00

## Overview of My Submission
A Wheel Of Fortune project that utilises REDIS in order to  maintain wheel segment prize allocation and distribution.

This project comes with a GRPC Server, Client and Admin client built in GO to maintain wheels.

A VUEJS website built with GRPC, ENVOY was created to spin the virtual Wheel Of Fortune!


## Submission Category: 
Wacky Wildcards

## Screenshots!

![Wheel Of Fortune vueJS app](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/atoeo4ttm73zhnpggu7z.png)


![Frontend - backend GRPC](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fzvtgz3hd80llwwwobse.png)

![Server listening!](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vh7fzrsfauz3r6q79m45.png)


![Wheel spins - CLI](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xow8dbsl2fqppg6a9p0g.png)


## Language Used
GO, Javascript, LUA

## Link to Code
{% embed https://github.com/jackmcguire1/wheeloffortune %}

## Additional Resources / Info

- REDIS for wheel of fortune management
- GO to expose a GRPC API
- ENVOY is used to provide a HTTP TO GRPC proxy
- VUEJS for front-end webapp


Jack McGuire [https://github.com/jackmcguire1](https://github.com/jackmcguire1)


* _Check out [Redis OM](https://redis.io/docs/stack/get-started/clients/#high-level-client-libraries), client libraries for working with Redis as a multi-model database._
* _Use [RedisInsight](https://redis.info/redisinsight) to visualize your data in Redis._
* _Sign up for a [free Redis database](https://redis.info/try-free-dev-to)._
Data Engineering 101: Introduction to Data Engineering
Tags: codenewbie, career, data
Published: 2022-08-21 18:32:46

**Data Engineering** is the process of building data pipelines and making quality data available for efficient data-driven decision-making.
A person who performs these activities is called a Data Engineer.

But what are data pipelines exactly...
In data processing, there is the flow of data from say a point A to B to C i.e., from an application to a data warehouse or from a data source to the database. This series of processing steps is called a **_data pipeline_**.
In these series of steps, each step delivers an output that is the input to the next step. This continues until the pipeline is complete. However, in some cases, independent steps may be run in parallel.


![Data Pipeline Patterns](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/43owm2xfucg3sq316gmv.png)

**What’s the difference between a data analyst and a data engineer?**
_Data scientists_ and _data analysts_ analyze data sets to gain knowledge and insights. _Data engineers_ on the other hand build systems for collecting, validating, and preparing that high-quality data which is then used by data scientists to promote better business decisions.

With that said, these are some of the **Essential skills required to be a Data Engineer in 2022**

- Data Structures
- SQL
- NoSQL
- Understanding of Data Lakes and Data Warehouse
- Python
- Big Data - Hadoop, Apache Spark(PySpark), Hive, and Apache Kafka
- Cloud Services - AWS, Microsoft Azure, Google Cloud, Snowflake, etc.
- Visualization - Tableau, PowerBI, Looker, Qlikview, etc.

I wish you all the best as you choose to pursue this journey.


**Thanks for reading!**
Any questions? Leave your comment below to start fantastic discussions!
Check hadith nabawi source and degree using Dorar API
Tags: reactnative, expo, react, api
Published: 2022-08-21 18:15:00


**Link to the webapp:**
✅ [Live demo](https://check-hadith.web.app)

**Link to the source code:**
✅ [Github](https://github.com/adelpro/check-hadith-native)

**With this application you can check:**
✅ The degree of hadith nabawi (sahih, makdoub ....)
✅ The source of the hadith with the page number
✅ The hadith teller


**Technologies used in this application:**
✅ [Dorar API](https://www.dorar.net/article/389/%D8%AE%D8%AF%D9%85%D8%A9-%D9%88%D8%A7%D8%AC%D9%87%D8%A9-%D8%A7%D9%84%D9%85%D9%88%D8%B3%D9%88%D8%B9%D8%A9-%D8%A7%D9%84%D8%AD%D8%AF%D9%8A%D8%AB%D9%8A%D8%A9-API) (modified to return a JSON result) thinks to [Ahmed El-tabarani](https://github.com/AhmedElTabarani/dorar-hadith-api) 
for the help
✅ EXPO
✅ React native
✅ Firebase (hosting)

**How To run the code?**
first :
`npm install`
then
`expo start`

**Support us ❤️**
[![Buymeacoffee](https://badgen.net/badge/icon/buymeacoffee?icon=buymeacoffee&label)](https://www.buymeacoffee.com/Adel.benyahia/)
Project: Nerd Crawler
Tags: art
Published: 2022-08-21 17:52:29

Nerd Crawler monitors 50+ original art sites and alerts you with an email or a text when new art drops for sale so you'll never miss out on one-of-a-kind original comic art ever again.

We integrate with top artist sites including Ed Benes, Jim Cheung, Mike Choi, and more! Plus, we monitor art dealer sites like Essential Sequential, Cadence Comic Art, and more!

Website: https://www.nerdcrawler.com/
Slowly move gameobject from one place to another in Unity3D
Tags: unity3d, csharp, gamedev, physi
Published: 2022-08-21 17:38:00

Sometimes we need to move a game object from one place to another and maintain the speed of movement. _Vector3.MoveTowards()_ can be the perfect solution for this. We need to create an empty gameobject and set the position where we need to move our gameobject. Then create another empty gameobject and make the previous two gameobject of it's child. 
Let's say, we need to move _DonkeyKong_ gameobject to the position of platformB. Then it can be like this.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pw1009a07thhvrb7ra5y.png)

Then, create a scripts, attach to the parent object and paste the following code.

`using System.Collections;
using System.Collections.Generic;
using UnityEngine;

public class PlatformMovement : MonoBehaviour
{
    private Vector3 pointA;
    private Vector3 pointB;
    Vector3 nextPos;
    [SerializeField] float speed;
    [SerializeField] Transform childTransform;
    [SerializeField]
    Transform transformB;
    // Start is called before the first frame update
    void Start()
    {
        pointA = childTransform.localPosition;
        pointB = transformB.localPosition;
        nextPos = pointB;
    }

    // Update is called once per frame
    void Update()
    {
        Move();
    }

    void Move()
    {
        childTransform.localPosition = Vector3.MoveTowards(childTransform.localPosition, nextPos, speed*Time.deltaTime);
        
    }

   
}
`

Then, set the variables to the inspector.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/yo7dymommjgadt6f1xu9.png)

If we want to move gameobject in a periodic motion, we need to add a couple of lines. 

`
using System.Collections;
using System.Collections.Generic;
using UnityEngine;

public class PlatformMovement : MonoBehaviour
{
    private Vector3 pointA;
    private Vector3 pointB;
    Vector3 nextPos;
    [SerializeField] float speed;
    [SerializeField] Transform childTransform;
    [SerializeField]
    Transform transformB;
    // Start is called before the first frame update
    void Start()
    {
        pointA = childTransform.localPosition;
        pointB = transformB.localPosition;
        nextPos = pointB;
    }

    // Update is called once per frame
    void Update()
    {
        Move();
    }

    void Move()
    {
        childTransform.localPosition = Vector3.MoveTowards(childTransform.localPosition, nextPos, speed*Time.deltaTime);
      //check if the position of gameobject is changed  if(Mathf.Abs(Vector3.Distance(childTransform.localPosition, nextPos) ) <= .1f)
        {
            ChangeDestination();
        }
    }

    void ChangeDestination()
    {
      //set the next position
        nextPos = nextPos != pointA ? pointA : pointB;
    }
}

`
Iniciando em Engenharia de dados
Tags: dataengineering, sql, python, beginners
Published: 2022-08-21 17:31:27

Cursos e livro que estudarei:

- [Bootcamp Engenharia de Dados - How](https://howedu.com.br/cohort/engenharia-de-dados/)
- [Data Engineer with Python - DataCamp](https://www.datacamp.com/tracks/data-engineer-with-python)
- [Intro to SQL - Kaggle](https://www.kaggle.com/learn/intro-to-sql)
- [Advanced SQL - Kaggle](https://www.kaggle.com/learn/advanced-sql)
- [SQL para Análise de Dados - Livro](https://www.novatec.com.br/livros/sql-para-analise-dados/)
Save 100+ hours with this document before building an app/website
Tags: showdev, app, webdev, beginners
Published: 2022-08-21 17:20:00

As an Indie developer, it is kind of easy & obvious to build anything without evaluating the pros n cons of the idea.

This is because, in product management process developers are tend to be on the solution side of the customer. But as an Indie you should see yourself how to work on the problem side of the customer too.

Before building, one should answer to the following questions:

1. What's your idea(one-liner)?
2. What's the opportunity size?
3. Who's your target audience?
4. What is/are the problem(s)?
5. What is/are the solution(s)?
6. How do you monetize it?
7. What are the other alternatives?

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kc4g84ugel0qo8uj71d2.png)
 


You can evaluate and manage your ideas in notion document
👉 [Download document](https://buildwithpradeep.gumroad.com/l/ideavaluation
)

----
checkout my previous post https://dev.to/pradeepb28_/showcase-avoid-those-awkward-moments-in-1-click-3gea
How to create a test database with Laravel Sail
Tags: tutorial, beginners, laravel, docker
Published: 2022-08-21 17:16:13

<!--
Title: How to create a test database with Laravel Sail
Tags: tutorial, beginners, laravel, sail, docker, database, testing
-->

## Introduction

I have worked on projects that required the production and testing database to be the same.

This could be because some features work in MariaDB, but not in SQLite. Or some bugs appear in MySQL, but not in PostgreSQL.

When you're working in a large project, **you need to know that the testing database works exactly like the one in production**.
You can't allow a bug to appear in production because the test database was different.

> Note: If the project is small, chances are you are fine using SQLite for testing.

Today, I'm going to guide you through setting up a testing database in your current database server, thanks to Laravel Sail.

---

## Before we start

This tutorial uses PostgreSQL as the database.
But the idea is the same for any other database: **create a separate database in your current database server**.

### Preface

### Requirements

- Laravel 9
- Laravel Sail

---

## Database setup

We will use the database server declared in the `docker-compose.yml` file.
In this case, PostgreSQL.

```yaml
services:
  pgsql:
    image: 'postgres:14'
```

First, enter PostgreSQL's CLI

```bash
sail psql
```

> Note: If you are using MySQL, you can use `sail mysql` instead.

Now, create a new database

```sql
CREATE DATABASE testing;
```

This database is separated from the default database that your application uses.

---

## PHPUnit configuration

Now, modify the `phpunit.xml` file to use the new database and database server.
Should end up similar to this:

```xml
<php>
    <env name="DB_CONNECTION" value="pgsql"/>
    <env name="DB_DATABASE" value="testing"/>
</php>
```

---

## Run the tests

Now, you can run the tests and see if they work.

```bash
sail test --parallel
```

It should look similar to this:

```bash
   PASS  Tests\Unit\ExampleTest
  ✓ example

  Tests:  1 passed
  Time:   0.7s
```

---

## End

That was it.
It's a one-time setup, that works like a charm.

<!-- INCLUDE -->

### Self-promotion

If you have found this useful, then you should follow me, I will be posting more interesting content! 🥰

- [GitHub](https://redirect.akbal.dev/github)
- [Twitter](https://redirect.akbal.dev/twitter)
- [Dev.to](https://redirect.akbal.dev/dev.to)

<!-- /INCLUDE -->

### Conclusion

Congratulations, today you have learned how to create a test database in Laravel Sail.
Without needing to modify the `docker-compose.yml` file or creating more services.

**Let me know if the tutorial was useful to you in the comments!**
Hand-on coding practice for frontend
Tags: webdev, javascript, react, css
Published: 2022-08-21 17:16:00



## Before you go for interview you must have handon coding practice using vanilla js or framework



**Here is the list** 👇

💥 **Beginner Level**

📌 Product Landing Page
📌 Javascript Music Player
📌 Calculator App
📌 Weather App
📌 Quiz App
📌 To Do List App
📌 Temperature Converter


💥 **Intermediate Level**

📌 Clone of Popular websites
📌 Expense Tracker App
📌 Book Finder App
📌 Dynamic Landing Page
📌 Web app using External API
📌 Card-Memory-Game
📌 Simple Online Store


💥 **Expert Level**

📌 Shuffle Card Deck
📌 Admin Dashboard including charts & graphs
📌 Blog or Ecommerce website
📌 Chat App
📌 Signin & Signup Forms using Authentication
📌 Survey App
Data Engineering 101
Published: 2022-08-21 16:51:22

Data engineering is a very distinct role in the field of IT and is at the top of the ranks in the data roles stack since it lays the foundation for the rest of the professionals like; business analysts, data analysts and data scientists. Precisely data engineering is the process of preparing data for data analysis and general consumption. This done through complex processes that ensure that the needed datasets are collected, transported and stored in the right place and can be accessed by consumers in the right format. Therefore, a data engineer is responsible for deciding the data to be collected, how to collect it, how to transfer and store it securely in the right format.


## Tools used in data engineering**
In the recent past big data has continued to grow immensely and it is projected to grow even further, recent projections show that by the year 2025 the world will be producing around 463 exabytes of data those are huge amounts of data, literally 18-digit amounts of data! This amount of data needs special skills and tools to be handled correctly, the commonly used tools and techniques include:

1. Programing using languages such as Python, Java, R, Scala etc. these languages are used to come up with tools such the data pipelines and for writing automation scripts.
2. Cloud computing technologies help in storing the data in large amounts as they can handle growing data. Examples of the providers include Microsoft Azure, Amazon web services and Google cloud platform. 
3. Database management systems that include SQL and NOSQL systems like MySQL, mongo DB etc. these systems are used to store and manipulate data.
4. Machine learning – though machine learning is mostly used by data scientists, data engineers need the skills so that they understand the needs of the data scientists so that they can serve them better.

## **Path to becoming a data engineer**

**A university degree **
 A university degree in computer science, mathematics and statistics, physics or any other related field is important but not mandatory employers look at the skill set and the ability to solve problems.

**Certifications **
There are a number of certifications that test specific skills and give certificates, most of these certificates are taught by industry experts and they are an added advantage in the job market. Some examples of platforms offering certifications include:
1.	Udacity  
2.	Coursera has a number of them
3.	Edx
4.	Udemy    
5.	Google
6.	Microsoft
7.	Aws 

**Bootcamps and communities **
These add value since they are made of people practicing data engineering and other related skills.
 
After acquiring the necessary skills what’s next? One needs to brand as data engineer in order to secure a job this can be done by creating an online portfolio and include projects that one has undertaken,  host them on personal websites or on GitHub.