teleren/devto huggingface dataset:
Using PySpark on AWS Glue Studio to load data from RDS MySQL to Data Lake
Tags: aws
Published: 2024-01-17 10:54:19

This post explores the powerful combination of PySpark and AWS Glue for streamlining data ETL (Extract, Transform, Load) processes. We'll delve into:

- PySpark: Harnessing Python's flexibility for large-scale data analysis and transformations within the familiar Python environment.

- AWS Glue: Simplifying and scaling ETL workflows with a fully managed, serverless service on AWS.

## The Challenge:

Efficiently transferring data from an RDS MySQL database to an S3 data lake.

## The Solution:

1. **Defining the ETL Job**: Moving data from `stats`, `ability`, and `info` tables in MySQL to S3.

2. **Setting Up Glue Studio**: Selecting `Author code with a script editor`, establishing IAM roles, and downloading the MySQL JDBC driver to S3.

3. **Coding with PySpark**: Utilizing the provided code template for a smooth workflow:
   - Creating a SparkSession.
   - Adding the JDBC driver.
   - Defining a function to extract data from tables.
   - Reading data from multiple tables.
   - Transforming the "capture_rate" in the "info" table.
   - Partitioning data into timestamp-based subfolders.
   - Writing data to S3 in Parquet format.

I have prepared for you code template for easier start with it:

```
from pyspark.sql import SparkSession
from pyspark.sql.functions import regexp_extract, col
from datetime import datetime

spark = SparkSession.builder.getOrCreate()  # Create SparkSession
spark.sparkContext.addPyFile("s3://<you-bucket>/mysql-connector-j-8.3.0.jar")  # Add the MySQL JDBC driver to the classpath
jdbc_url = "jdbc:mysql://<your-host>:3306/<your-database>"
connection_properties = {"user": "admin", "password": "********"}

def extract_df_to_s3(spark_session, jdbc_url, connection_properties, table_name):
    df = spark_session.read.jdbc(url=jdbc_url, table=table_name, properties=connection_properties)
    return df

# Read data from multiple tables
df_info = extract_df_to_s3(spark, jdbc_url, connection_properties, "info")
df_ability = extract_df_to_s3(spark, jdbc_url, connection_properties, "ability")
df_stats = extract_df_to_s3(spark, jdbc_url, connection_properties, "stats")

# Transform capture_rate in the info table and cast to int
df_info = df_info.withColumn("capture_rate", regexp_extract("capture_rate", r"^\D*(\d+).*$", 1)) \
                 .withColumn("capture_rate", col("capture_rate").cast("int"))

# Generate timestamp subfolders for each DataFrame
timestamp_str = datetime.now().strftime("%Y-%m-%d_%H-%M-%S")
info_subfolder = f"pokemon_info_processed/{timestamp_str}"
ability_subfolder = f"pokemon_ability_processed/{timestamp_str}"
stats_subfolder = f"pokemon_stat_processed/{timestamp_str}"

# Write DataFrames to separate folders
df_info.write.parquet(f"s3://<your-bucket>-datalake/{info_subfolder}", mode="overwrite")
df_ability.write.parquet(f"s3://<your-bucket>-datalake/{ability_subfolder}", mode="overwrite")
df_stats.write.parquet(f"s3://<your-bucket>-datalake/{stats_subfolder}", mode="overwrite")

```

Now you can run the job and check if the result appear in the S3 Data Lake bucket. ✨
Reinforcement Learning in 5 Minutes
Tags: ai, programming, webdev, python
Published: 2024-01-17 10:51:44

We want our AI model to make right decisions, and one powerful approach to achieve this is through reinforcement learning. Reinforcement Learning is a subfield of machine learning that involves training an agent to make decisions by selecting actions from its action space within a specific environment, with the goal of maximizing cumulative rewards over time.


{% youtube 2ArU1uE9QLo %}

## How Do We Define Reinforcement Learning?
Reinforcement Learning is a subfield of machine learning that teaches an agent how to choose an action from its action space, within a particular environment, in order to maximize rewards over time.

## Reinforcement Learning has four essential elements:

- Agent. The program you train, with the aim of doing a job you specify.
- Environment. The world, real or virtual, in which the agent performs actions.
- Action. A move made by the agent, which causes a status change in the environment.
- Rewards. The evaluation of an action, which can be positive or negative.

## Supervised, Unsupervised, and Reinforcement Learning: What are the Differences?

## Difference #1: Static Vs.Dynamic
The goal of supervised and unsupervised learning is to search for and learn about patterns in training data, which is quite static. RL, on the other hand, is about developing a policy that tells an agent which action to choose at each step — making it more dynamic.

## Difference #2: No Explicit Right Answer
In supervised learning, the right answer is given by the training data. In Reinforcement Learning, the right answer is not explicitly given: instead, the agent needs to learn by trial and error. The only reference is the reward it gets after taking an action, which tells the agent when it is making progress or when it has failed.

## Difference #3: RL Requires Exploration
A Reinforcement Learning agent needs to find the right balance between exploring the environment, looking for new ways to get rewards, and exploiting the reward sources it has already discovered. In contrast, supervised and unsupervised learning systems take the answer directly from training data without having to explore other answers.

## Difference #4: RL is a Multiple-Decision Process
Reinforcement Learning is a multiple-decision process: it forms a decision-making chain through the time required to finish a specific job. Conversely, supervised learning is a single-decision process: one instance, one prediction.

Reference: https://www.epicprogrammer.org/reinforcement-learning/
To fetch the unique characters from a string?
Published: 2024-01-17 10:48:05

```
using System;
using System.Linq;
public class HelloWorld
{
    public static void Main(string[] args)
    {
        string s = "AAABBBBBCCCCFFFFGGGGGDDDDJJJJJJ";
        var newstr = String.Join("", s.Distinct());
        Console.WriteLine (newstr);
    }
}
```
Decoding Weird JavaScript Behaviour
Tags: webdev, javascript, learning
Published: 2024-01-17 10:45:05

## History

JavaScript was created by Brendan Eich in 1995 while he was working at Netscape.

The initial version of JavaScript was developed in just 10 days in May 1995.

Browsers primarily understand and interpret three core languages for web development: HTML, CSS & JavaScript (not even TypeScript 🙂).

JavaScript is a versatile programming language used both on the client side and increasingly on the server side (Node JS).

He is also the founder of two famous browsers: Mozilla Firefox and Brave.

## Type Coercion

One of the most common sources of weird behaviour in JavaScript is type coercion.

Coercion in simple terms is the process of automatically converting values from one type to another type.

This is one of the reasons why “type” is important.

![Type-Coercion](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2fnr6ns95g23c8i9lfr0.png)

## Floating Point Numbers

Floating-point numbers are used to represent decimal numbers in JavaScript.

They have limited precision due to their fixed bit-size (64 bits in JS).

Solutions:

- Use a library like Decimal.js for precise calculations.
- Use the toFixed() method to round floating-point numbers to a certain number of decimal places.

![Floating-Point-Numbers](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/yod58pt8z9ymhfyaqrp0.png)

## Equality

JavaScript has two equality operators right now: == and ===.

The == operator checks for equality regardless of type, while the === operator checks for equality and type.

To avoid these unexpected results, you should always use the === operator when comparing values.

![Equality](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8md76ycven391zs2dcdg.png)

## undefined vs. null

Undefined: It represents a variable that has been declared but not assigned a value.

Null: It represents an intentional absence of any object value.

The == operator treats undefined and null as equal, but the === operator does not.

![Undefined-null](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jjrxhix09aeej26asbsf.png)

## NaN

NaN stands for "Not a Number"

It is a special value that represents an invalid numerical value.

The isNaN() function in JavaScript is used to check if a value is NaN

In PHP, you have the is_nan() function to check if a value is NaN.

![nan](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7x90ohhjno4pmwldhdcm.png)

That's it for this article. I hope you must have learnt something new from this article or may be now you know the reason behind the weird behaviour of JavaScript. I will also make part 2 of this very soon. So stay tuned. Thank you.
Ultimate Android Fitness Watch Your Ideal Workout Companion
Published: 2024-01-17 10:38:59

In today's fast-paced world, staying fit and healthy is a top priority for many individuals. Thanks to advancements in technology, achieving your fitness goals has become more convenient and personalized. If you're an Android user on the lookout for the perfect fitness companion, you're in the right place. In this blog, we'll explore the features and benefits of an Android fitness watch without diving into any specific brand names.

## Why Choose an Android Fitness Watch?
Android fitness watches have become an indispensable tool for fitness enthusiasts, providing a seamless integration between technology and wellness. These smartwatches are designed to work harmoniously with Android devices, ensuring a user-friendly experience for those who are deeply invested in the Android ecosystem.
Key Features to Look for in an Android Fitness Watch

### 1. Compatibility
The foremost consideration when choosing an **[Android fitness watch](https://getlifewatch.com/products/2-deluxe-life-watches)** is its compatibility with your Android smartphone. Ensure that the watch seamlessly connects with your device, allowing for effortless synchronization of data.

### 2. Health and Fitness Tracking
Look for a watch that goes beyond basic step counting. The ideal Android fitness watch should offer comprehensive health and fitness tracking features, such as heart rate monitoring, sleep analysis, and GPS tracking for accurate distance and location data.

### 3. Battery Life
Long battery life is a key consideration for any wearable device. Choose an **[Android fitness watch](https://getlifewatch.com/products)** that can keep up with your active lifestyle without constantly needing to be charged.
**4. Water Resistance**
For those who engage in water-based activities or simply want a watch that can withstand sweat and rain, water resistance is a must. Look for a watch with a suitable water resistance rating to ensure durability.

**The Android Fitness Watch Advantage**
Android fitness watches offer a plethora of benefits for users who are committed to their health and well-being. By seamlessly integrating with Android devices, these watches provide a holistic approach to fitness tracking and ensure a user-friendly experience.
Whether you're a seasoned fitness enthusiast or just starting your wellness journey, an Android fitness watch can be a valuable companion. Choose a watch that aligns with your preferences, lifestyle, and fitness goals, and embark on a journey to a healthier, more active you.

**In conclusion,**
the world of **[Android fitness watches](https://getlifewatch.com/products/2-deluxe-life-watches)** is vast, offering a range of features to cater to diverse user needs. Finding the right watch for you involves considering factors like compatibility, tracking capabilities, battery life, and more. With the right Android fitness watch by your side, you'll be well-equipped to conquer your fitness goals and live a healthier, more active life.
Time-Travel Debugging: A Playwright Traces Tutorial
Tags: playwright, testing, javascript, node
Published: 2024-01-17 10:32:29

**Ever wished you could replay a test that failed in your CI?** That's where Playwright Traces come in! This guide shows how it seamlessly fits into your CI workflow, making bug-hunting easier and more effective.

## Debugging Trace in Action

Let's jump right into a practical scenario. In the video below, I'll show how to use the Playwright Trace Viewer to inspect a failed test, identify the issue, and apply a fix.

{% youtube MTgNbdG6I3A %}

In this video, I focus on resolving a complex locator issue. Besides this, Playwright Trace Viewer offers an extensive array of features for pinpointing test failures, including network logs, and console messages or step-by-step snapshots, not covered here.

## Integrating with Your CI

### 1. Record Traces when a Test Fails

To automatically record traces on your CI, simply update your Playwright config file with this option:

```ts
use: {
  trace: 'on-first-retry',
}
```

Now, whenever a test fails, a trace file is generated, ready for download or online viewing with a third-party reporter.

### 2. Open Trace Files

To open a Trace file, either upload it to [trace.playwright.dev](https://trace.playwright.dev) or run the command:

```sh
npx playwright show-trace <TRACE_FILE_PATH>.zip
```

### 3. Third-Party Reporting with Argos

To enhance your workflow, you can integrate your CI with [Argos](https://argos-ci.com) for direct access to Playwright Traces online. This seamless connection eliminates the need to download trace files, offering a more streamlined approach.

Check out the [Argos Playwright setup guide](https://argos-ci.com/docs/quickstart/playwright) for easy integration steps.

![Github fail test](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rtfqg293cx1z2ebpd5kr.png)

![Argos with Trace](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8ec6ynjjslthxw7jznai.png)

## Further Reading

- [Playwright Trace Viewer Documentation](https://playwright.dev/docs/trace-viewer)
- [Argos Documentation](https://docs.argos-ci.com)
- [Debbie O'Brien's Video on Playwright Traces](https://www.youtube.com/watch?v=lfxjs--9ZQs) by [Debbie O'Brien](https://twitter.com/debs_obrien)
Beli Desain di Marketplace Desain jjpedia
Tags: design, creative, bisnis
Published: 2024-01-17 10:25:06

Kini ada cara mudah untuk mendapatkan desain kreatif dengan harga murah! Cocok untuk usaha UMKM ataupun Anda yang sedang ingin merintis usaha dan ingin produknya dilihat secara digital!

Tempatnya adalah jjpedia UMKM! Disini hadir website marketplace [jual desain](https://jjpedia.com/umkm) murah khusus untuk Anda para pelaku UMKM di seluruh Indonesia!

Kunjungi jjpedia UMKM dan dapatkan desain-desain murah untuk mengiklankan ataupun menampilkan bisnis atau produk Anda.
Freelancer Survey 2024
Tags: discuss, news, community
Published: 2024-01-17 10:11:11

Hi there 👋,

I would like to let you know that freelancermap.com has launched the 2024 Freelancer Survey this week, and all freelancers are kindly invited to participate and share their experiences and insights: https://freelancermap.eu.typeform.com/survey-2024?utm_source=devcommunity

🙏
Odoo : configure Debugger in Pycharm
Tags: odoo
Published: 2024-01-17 10:09:08

**Important Note**: This tutorial focuses on Odoo development using the source code folder, as opposed to the Docker approach.

Prerequisites:

-     Install PyCharm.
-     Place the Odoo source code folder.

Open the PyCharm application either from the UI or command line. Once PyCharm is launched, navigate to the Main Menu items > Run > Edit Configurations. This action might open a window as depicted below. If not, simply click the + button, and a new default dialogue window will appear.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ktkexgegqhg75k6fbgz2.png)

In the default new configuration, the script parameter box may not be visible. Click on the "modify options" and select "parameters."

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/iakk63vs3i6kt77auh2v.png)

Once all the required values are entered, close the window. A debug icon will appear on the top side of the window, as shown in the screenshot (the green bug).


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5frsn5ylui5mr2hb0pu5.png)

Next, set a breakpoint in your code and run Odoo with the debugger. It will hit the breakpoint, and the debugger console will be displayed. Python code can be executed in the debug console to check any expressions.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/64gqo58r6eom607bfcap.png)

Once in the debugger console, you can use the following code to gather more information about the model and related components:

```
self.env['sale.order'].search([])
self.env.context

```
 
The first line (self.env['sale.order'].search([])) allows you to search for any model file, where 'sale.order' represents the model file in this example. The second line (self.env.context) provides information about the current window/page in the Odoo UI.
The role of a web design company Singapore:-Subraa
Tags: web, design, company, singapore
Published: 2024-01-17 10:01:02


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8gutt3oal6jnx0aqelw7.jpg)

Demand for web design services underscores its enduring relevance. As businesses strive to establish a robust online presence, the role of a [web design company Singapore](https://www.subraa.com/) remains paramount for a multitude of reasons.

**1. Digital First Impressions:**
A website is often the first interaction a potential customer has with a brand. A well-designed website serves as a digital storefront, shaping the initial impressions that significantly influence user perceptions.

**2. Evolving Aesthetics and Functionality:**
Web design is not static; it evolves with design trends and technological advancements. Businesses seek the expertise of web design companies to ensure that their websites align with current aesthetics, providing a contemporary and engaging user experience.

**3. Responsive Design Imperative:**
With the surge in mobile users, responsive design has become imperative. Web design companies excel in creating websites that seamlessly adapt to various devices, ensuring an optimal user experience regardless of the platform.

**4. User Experience and Navigation:**
Beyond aesthetics, user experience is critical. Web design companies specialize in creating intuitive navigation structures and user interfaces, enhancing the overall experience for visitors and translating their journeys into meaningful interactions.

**5. SEO Integration:**
The intricacies of search engine optimization (SEO) are integral to a website's success. Web design companies integrate SEO best practices during the design phase, contributing to improved visibility on search engines and heightened discoverability.

**6. Brand Consistency:**
A cohesive brand presence across digital platforms is vital. Web design companies ensure that the website aligns with the overall brand identity, fostering consistency that strengthens brand recall and trust.

**7. Technological Expertise:**
Staying abreast of technological advancements is a web design company's forte. From incorporating interactive elements to implementing the latest security features, their technological expertise ensures that websites are not only visually appealing but also functionally robust.

**8. E-Commerce Evolution:**
The rise of e-commerce has heightened the demand for effective and visually appealing online stores. Web design companies play a pivotal role in creating e-commerce platforms that not only showcase products effectively but also facilitate seamless transactions.

In conclusion, the demand for web design services emanates from its multifaceted role in shaping digital experiences. Beyond aesthetics, it encompasses functionality, user experience, and the dynamic nature of the online landscape, solidifying its position as a cornerstone in the digital success of businesses across diverse industries.

**Visit Our Site: https://www.subraa.com/**
Wednesday Links - Edition 2023-01-17
Tags: java, jvm, loom, docker
Published: 2024-01-17 10:00:00

Managing system overload (10 min)⛈️
https://blog.softwaremill.com/navigating-through-the-storm-93c54fa44f38

JEP 447: Refining Java Constructors for Enhanced Flexibility (2 min)🦵
https://www.infoq.com/news/2024/01/java-constructors-enhancement/

Virtual Thread Pinning - What it is, how to detect it, and how to avoid it (6 min)📌
https://todd.ginsberg.com/post/java/virtual-thread-pinning/

Hello eBPF: Developing eBPF Apps in Java Part 1 (6 min)🌰
https://mostlynerdless.de/blog/2023/12/31/hello-ebpf-developing-ebpf-apps-in-java-1/

Hard disk LEDs and noisy machines (4 min)🚦
https://blogsystem5.substack.com/p/hard-disk-leds-and-noisy-machines

Kicking the tires of Docker Scout (2 min)🛞
https://blog.frankel.ch/kicking-tires-docker-scout/

Statistics for the 1BRC Java Challenge (2 min)📊
https://minborgsjavapot.blogspot.com/2024/01/statistics-for-1brc-java-challenge.html

Oracle Database 23c (30 sec)🏢
https://twitter.com/aalmiray/status/1745923133289181423

IntelliJ HTTP Client: Allowing Insecure HTTPS Requests (1 min)🔓
https://blog.mrhaki.com/2024/01/intellij-http-client-allowing-insecure.html

Do we think of git commits as diffs, snapshots, and/or histories? (8 min)📜
https://jvns.ca/blog/2024/01/05/do-we-think-of-git-commits-as-diffs--snapshots--or-histories/
CSS: Punching Through the Surface
Tags: css, frontend, design, html
Published: 2024-01-17 09:59:06

While working on [Cobhams Player](https://cobhams.netlify.com), I needed to design a surface that gives an impression that it's been punched through. I expected the implementation of this effect to be quite straightforward in CSS. Turns out I was wrong for my use case. I ended up with my own implementation. However, I came across a certain CSS property which has proven useful in this regard: `background-attachment`. Let's see how it works.

```html
<div class="container">
   <div class="surface">
       <div class="hole"></div>
   </div>
</div>
```
Our HTML code has a main `container`, a `surface` that we need to "punch through" and the `hole` we're punching. And here's our CSS code:
```css
.container {
    padding: 28px;
    width: 100%;
}

.surface {
    display: flex;
    justify-content: center;
    align-items: center;
    background-color: black;
    width: 300px;
    height: 300px;
    margin: auto;
}

.hole {
    height: 150px;
    width: 150px;
    border-radius: 50%;
}
```

![CSS Hole](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/i1yelo4o0jimfwo5o0pd.png)

As you can see, our black surface is yet to show a hole, so let's make it happen. It's quite straightforward if our container's background has a plain color, say blue. We only need to add the same background to the hole:

```css
.container {
    padding: 28px;
    width: 100%;
    background: blue; // <-- add this
}

.hole {
    height: 150px;
    width: 150px;
    border-radius: 50%;
    background: blue; // <-- add this
}
```


![CSS Hole](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3x016fkn1x9jq4bs99pa.png)

But what happens if our surface has say a gradient background?

![CSS Hole](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0a5tqau2ohz2uojslzb1.png)

So can we solve this by adding the same gradient to the hole?

![CSS Hole](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7umt3qiy8zp3t4du6l11.png)

Looks better right? But it doesn't really look much like a hole through to the background. This is where `background-attachment: fixed;` can be of help:

```css
.hole {
    height: 150px;
    width: 150px;
    border-radius: 50%;
    background: linear-gradient(239deg,#de2134 27.11%,#38bf8e 73.91%);
    background-attachment: fixed; // <-- Add this
}
```

![CSS Hole](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/v6j37b8yfds2mle0zl9j.png)

That's it. Now we've successfully created a hole that punches through. 

Thank you for reading. Let me know what you think.