teleren/devto huggingface dataset:
Spark on AWS Glue: Performance Tuning 2 (Glue DynamicFrame vs Spark DataFrame)
Tags: aws, glue, spark, performance
Published: 2023-07-16 21:26:01


This is a continuation of my previous posts as follows.

1. [Spark on AWS Glue: Performance Tuning 1 (CSV vs Parquet)](https://dev.to/tmyoda/spark-on-aws-glue-performance-tuning-1-csv-vs-parquet-2l5)

2. [Spark on AWS Glue: Performance Tuning 2 (Glue DynamicFrame vs Spark DataFrame)](https://dev.to/tmyoda/spark-on-aws-glue-performance-tuning-2-glue-dynamicframe-vs-spark-dataframe-12gb)

3. [Spark on AWS Glue: Performance Tuning 3 (Impact of Partition Quantity)](https://dev.to/tmyoda/spark-on-aws-glue-performance-tuning-3-impact-of-partition-quantity-4i55)

4. [Spark on AWS Glue: Performance Tuning 4 (Spark Join)](https://dev.to/tmyoda/spark-on-aws-glue-performance-tuning-4-spark-join-39m3)

5. [Spark on AWS Glue: Performance Tuning 5 (Using Cache)](https://dev.to/tmyoda/spark-on-aws-glue-performance-tuning-5-using-cache-4ob5)


## Glue DynamicFrame vs Spark DataFrame
Let's compare them using the Parquet file which I created in the part 1.

### Data Read Speed Comparison

We will read a single large Parquet file and a highly partitioned Parquet file.

```python
with timer('df'):
    dyf = glueContext.create_dynamic_frame.from_options(
        "s3",
        {
            "paths": [
                "s3://.../parquet-chunk-high/"
            ]
        },
        "parquet",
    )
    print(dyf.count())
    
with timer('df partition'): 
    dyf = glueContext.create_dynamic_frame.from_options(
        "s3",
        {
            "paths": [
                "s3:/.../parquet-partition-high/"
            ]
        },
        "parquet",
    )
    print(dyf.count())
```
```
324917265
[df] done in 125.9965 s
324917265
[df partition] done in 55.9798 s
```

DynamicFrame is too slow...

### Summary
- Based on the part 1 (Reading Speed Comparison), spark.read is 27.1 s (for single large file) and 36.3 s (for highly partitioned file), so DynamicFrame is quite slow.
- Interestingly, the speed of reading partitioned data is faster than single large Parquet file.
Python Flask REST API Generator...
Tags: python, flask, windev, windows
Published: 2023-07-16 21:05:09

## Scoriet, a program generator
Hi community,

long time ago, I thought about a possibility to extend my code generator, to generate python code too. It's finally here, hopefully it's of use to the community.

[Link to scoriet.com](https://www.scoriet.com)

What you get is an open-source [Github Project](https://github.com/Harveyhase68/scoriet) for reading a mysql or postgresql database and generate a whole app based on a template. It is a Windows 32-bit or 64-bit program, running Windows 7+.

Initially developed for my own purposes, only to generate PHP snippets or small classes.

But time comes and I decided to expand the tool, later even make it open-source.

Last week, I talked a lot on discord, but there was no interest in a python rest API generated by a windows open-source program...

So here I am, presenting at dev.to, maybe here are some developers who desperatly need such a tool??

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/yz7kvx7biltol1xvgfvx.png)

So, it connects to a mysql or a postgresql database, import the structure with constraints and keys, then it can generate a complete app with a mouse click, based on a template. You can stack some compatible templates, make changes to stock templates and more. You can export / import new/change, share templates with other users and more.

You can import a database (this is version 1), you make changes to the database, import again (this is version 2), every version is stored, so you can, for example, make a model (for laravel as example), to make changes to the latest database model?!

Actual, I made a simple web-site, a PHP REST API and now a Python Flask REST API.

When I have more time, maybe I can even finish the database designer (for new or editing existing databases), this is my next project for Scoriet...

I have a small developer blog ([www.predl.cc](https://www.predl.cc/entwicklerblog)) but actual in German, later it will be extended to English!

Comments, wishes and criticism are welcome, hope this project can be a help for you. 

PS: not every change on the template I have to do, you can do it on yourself?! Look at the document folder scoriet/templates/python_rest you find there the template files, open with any editor (notepad++ is ok), but not with Microsoft Studio Code, because, the template keys are with curly brackets for example {filename} and Microsoft Studio Code don't like them! But if you do changes and want to share with the project, please send me the files or complete templates to office[add]scoriet.com, I'll check them and include them in the setup.
Very Useful Algorithm SASS
Tags: frontend, webdev, css, beginners
Published: 2023-07-16 20:43:06

Hi. It won't be lengthy!⏰
## Short introduction
I'm assuming you've run into a problem where you needed to include all your fonts in a css file and it was really repetitive and boring. So, I developed a SASS algorithm that will help you in such a situation. This is a very simple algorithm but still useful.
## How to use
To use it properly you just have a folder where your fonts are separated by their names and each font has its own folder like this:
![folder](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/r34o93qfqaxsihn4bje3.png "Fonts")
You can then paste this code into your SASS file:
```scss
$path-to-fonts: '../fonts';
$font-families: (
    'Font1': ((100, 'Thin'),
        (200, 'ExtraLight'),
        (300, 'Light'),
        (400, 'Regular'),
        (500, 'Medium'),
        (600, 'SemiBold'),
        (700, 'Bold'),
        (800, 'ExtraBold'),
        (900, 'Black')),
    'Font2': ((100, 'Thin'),
        (200, 'ExtraLight'),
        (300, 'Light'),
        (400, 'Regular'),
        (500, 'Medium'),
        (600, 'SemiBold'),
        (700, 'Bold'),
        (800, 'ExtraBold'),
        (900, 'Black')),
    'Font3': ((100, 'Thin'),
        (200, 'ExtraLight'),
        (300, 'Light'),
        (400, 'Regular'),
        (500, 'Medium'),
        (600, 'SemiBold'),
        (700, 'Bold'),
        (800, 'ExtraBold'),
        (900, 'Black'))
);

@each $font-family,
$weights in $font-families {
    @each $weight in $weights {
        $weight-number: nth($weight, 1);
        $weight-name: nth($weight, 2);

        @font-face {
            font-family: $font-family;
            font-style: normal;
            font-weight: $weight-number;
            src: url('#{$path-to-fonts}/#{$font-family}/#{$font-family}-#{$weight-name}.ttf') format('truetype');
        }
    }
}
```

Instead of `font1`, `font2` and `font3`, you can manipulate this object as you wish and give your own font names, so instead of `"Font1"` give the name of your font, for example `"Roboto"`. Also you can edit, add or remove font weights you have but use the pattern `(WeightNumber, 'WeightName')` for example `(400, 'Regular')`

You can change the path to your fonts, add or remove a font style by adding a new block of code, eg.

```scss
@font-face {
            font-family: $font-family;
            font-style: YourFontStyle;
            font-weight: $weight-number;
            src: url('#{$path-to-fonts}/#{$font-family}/#{$font-family}-#{$weight-name}Italic.ttf') format('truetype');
        }
```
Here `'YourFontStyle'` can be normal, italic, oblique etc.

> If you want to look at the full example and you can also copy the code from [here](https://gist.github.com/nikitadmitr/0e5c1b102273ab4895b3f84dd5ea4b3d). If you have any ideas it would be great if you contribute!
i can't resolv core policy issue on my laravel project any help
Tags: laravel
Published: 2023-07-16 20:34:08


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/l777flt6xq071rzsf2zr.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/yxpeofgausamnjlo41rk.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/keehpkbljuisze1ea0ok.png)
Code Smell 218 - Magic Concatenation
Tags: webdev, beginners, programming, python
Published: 2023-07-16 20:30:39

*If you miss a comma, you are concatenating*

> TL;DR: Watch out for fancy language assumptions

# Problems

- Possible defects

- The [least surprise principle](https://en.wikipedia.org/wiki/Principle_of_least_astonishment) violation

- Hidden Assumptions

# Solutions

1. Be declarative

2. Use good linters

3. Prefer declarative languages

# Context

Many programming languages favor laziness over readability and clean code.

You should use them with caution and trust your tests.

# Sample Code

## Wrong

[Gist Url]: # (https://gist.github.com/mcsee/0f9cad84530ebad6592fa17cb016f2c8)
```python
tools = [
    "Amazon Codewhisperer",
    "Bard" # Notice the missing comma
    "ChatGPT",
    "Dalle-E" # Also here
    "Eliza"
    ]
    
print(len(tools))
# This will output 3 

print(tools)
# ['Amazon Codewhisperer', 'BardChatGPT', 'Dalle-EEliza']
# Missing Commas act as hidden string concatenators


```

## Right

[Gist Url]: # (https://gist.github.com/mcsee/4dfc8f7fb9189f1aa2f2986e924f41d1)
```python
tools = [
    "Amazon Codewhisperer",
    "Bard",
    "ChatGPT",
    "Dalle-E",
    "Eliza"
]

# We added all the missing commas

print(len(tools))
# 5

print(tools)
# ['Amazon Codewhisperer', 'Bard', 'ChatGPT', 'Dalle-E', 'Eliza']
```

# Detection

[X] Semi-Automatic 

Many linters warn about this problem. 

Also, [ChatGPT](https://chat.openai.com/) and [Bard](https://bard.google.com/) can detect the problem.

# Tags

- Readability

# Conclusion

Many modern programming languages come with a significant amount of accidental complexity. 

They are often optimized for writing code quickly, even though they may be prone to defects.

Unfortunately, when working with these languages, it is essential to exercise extreme caution.

# Relations

{% post https://dev.to/mcsee/code-smell-84-max-min-javascript-d39 %}

{% post https://dev.to/mcsee/code-smell-69-big-bang-javascript-ridiculous-castings-49ij %}

{% post https://dev.to/mcsee/code-smell-06-too-clever-programmer-37lb %}

# Disclaimer

Code Smells are my [opinion](https://dev.to/mcsee/i-wrote-more-than-90-articles-on-2021-here-is-what-i-learned-1n3a).

# Credits

Photo by [Edge2Edge Media](https://unsplash.com/@edge2edgemedia) on [Unsplash](https://unsplash.com/photos/t1OalCBUYRc)
    
* * *

> A programming language is low level when its programs require attention to the irrelevant.

_Alan J. Perlis_
 
{% post https://dev.to/mcsee/software-engineering-great-quotes-26ci %}

* * *

This article is part of the CodeSmell Series.

{% post https://dev.to/mcsee/how-to-find-the-stinky-parts-of-your-code-1dbc %}

* * *

My new book about clean code is available for pre-order.

You will find several recipes like this one with a higher level of detail

[![Book](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/oe6rgdrtmlohk9bmvfyb.jpg)](https://amzn.to/44s1XdO)
Experiments with eBPF: Snitchrs
Tags: ebpf, linux, rust, kubernetes
Published: 2023-07-16 20:23:20

In this post, I will share my experience on learning and playing with Extended Berkeley Packet Filter (eBPF). It's a very cool technology that not anyone might be aware of. It's been a while since I planned to learn some eBPF and this post has some good information for anyone who wants to get started. eBPF is a Linux kernel feature that allows to easily and safely interact with the kernel.

In order to learn how to use eBPF, I decided to implement Snitchrs a simple program that shows on a map to which IPs your computer is connecting to. 

I will go through interesting snippets of code I had to write in order to implement this program. This article is not meant to be a tutorial but rather an explanation that will hopefully help you better understand useful things to know in order to write your own eBPF programs. I'll assume some basic knowledge of the TCP/IP stack and how the Linux kernel works. 

[Continue reading... ](https://blog.fponzi.me/2023-06-22-experiments-with-ebpf-snitchrs.html) but feel free to leave comments here!
Python Table Console
Published: 2023-07-16 20:21:59

---
title: Python Table Console
published: true
description: 
tags: 
# cover_image: https://direct_url_to_image.jpg
# Use a ratio of 100:42 for best results.
# published_at: 2023-07-16 20:13 +0000
---

## pip install prettytable 



from prettytable import PrettyTable
 
# Specify the Column Names while initializing the Table
myTable = PrettyTable(["Student Name", "Class", "Section", "Percentage"])
 
# Add rows
myTable.add_row(["Leanord", "X", "B", "91.2 %"])
myTable.add_row(["Penny", "X", "C", "63.5 %"])
myTable.add_row(["Howard", "X", "A", "90.23 %"])
myTable.add_row(["Bernadette", "X", "D", "92.7 %"])
myTable.add_row(["Sheldon", "X", "A", "98.2 %"])
myTable.add_row(["Raj", "X", "B", "88.1 %"])
myTable.add_row(["Amy", "X", "B", "95.0 %"])
 
print(myTable)



## pip install tabulate 

# import module
from tabulate import tabulate

# assign data
mydata = [
	["Nikhil", "Delhi"],
	["Ravi", "Kanpur"],
	["Manish", "Ahmedabad"],
	["Prince", "Bangalore"]
]

# create header
head = ["Name", "City"]

# display table
print(tabulate(mydata, headers=head, tablefmt="grid"))
Static 2 level menu in Go
Tags: webdev, go, nav
Published: 2023-07-16 20:15:00

**Navigation in General**
When it comes to designing navigation menus, it's generally recommended to keep the levels of depth to a minimum for optimal user-friendliness. 

In my experience, a maximum of two levels deep provides a user-friendly navigation experience and reduces the chances of confusion or getting lost. 

A mobile-first approach ensures that the navigation remains user-friendly across different devices and screen sizes.

**Using Go HTML Templates**
By using Go HTML templates, you can simplify the process of creating navigation menus. Go templates are a powerful feature of the Go language that allows for the creation of HTML pages. 

Instead of hard-coding navigation elements into each individual page, you can define a single navigation sub-template that is shared across the entire site. 

Go templates provide many functions that enhance the flexibility of navigation implementation. Templating functions like "if" enable you to conditionally render navigation elements based on parameters. 

[In this micro guide](https://nav.go4webdev.org), I will show you how I created a static two-level menu navigation using Go templates.
microsoft teams -app
Published: 2023-07-16 20:04:23

Hi @all, 
In our organization, we have created apps to take people poll/option of getting daily option for food, like below 

Team app will do,

STEP1: user need to send hi to app

STEP2: app will reply with Yes/No to select food for tommorow

STEP3: user need to reply by selecting yes/no in the chart.

I would like to automate this reply to this app which is created by our org.

Could you confirm me in this how can i create the app or make auto reply to the app?

Thanks!!
innerHTML, outerHTML and other properties #20
Tags: javascript, java, programming, beginners
Published: 2023-07-16 19:32:15

// Chapter 8 Event & Other DOM Proterties 
// console.dir founction 
// console.log show the element as an element iwth its properties 

// tag name / node Name 
// Used to read the tag of an element 
// tagname --> only inisist for element node 
// nodename --> defined for any node (text , comment ,etc)

// Innter HTML and Outer HTML 
// The inner HTML property allow to get the HTML inside the element as a string **void for the element node only *****

// The outer HTML property contains the full HTML inner HTML + the element itselft 

// inner HTML is valid only for elements nodes . For other node type we can use nodevalue or data 

// text content 
// provide acces to the text inside the element , only txt ,minus all tags 

// The Hidden Property 
// The hidden attribute and the DOM property specfies wheather the element is visible or not 

let x = document.getElementsByTagName('span')[0]
console.log(x)
let y = document.getElementsByTagName('span')[0]
console.dir(y)
console.log(document.body.firstChild.nodeName)
console.log(document.body.firstElementChild.nodeName)

// first.innerHTML 
// first.innerHTML = "<i>hey I am italic</i>" 
// first.outerHTML 
// first.outerHTML = "<div>hey</div>"
// document.body.firstChild.data 
// console.log(document.body.textContent)
// first.hidden = false
French - AF - Day 8
Published: 2023-07-16 18:38:08

<u>**_# Sentences used in the CLASS_**</u>

1. Je suis malade.
2. C'est vendredi 
3. le 14 jullet
4. La Fête nationale
5. le premier ministre
6. Quoi de spécial de weekend ?
7. les vacances
8. Voyager
9. Je ne sais pas
10. rien de spécial
11. vous êtes triste
12. où - where
13. allez au cinéma
14. il pleut ici
15. n'est pas masculin
16. Regarde ici
17. juste la première catégorie 
18. Quelle type de verb 'er' ?
19. comment conjuguer le verbe ?
20. dites moi - Tell me
21. quelle est la prononciation?
22. je vais dire en anglais - i will say in english
23. lisez - read
24. Qui habite Paris ? Parisien
25. on va écouter - we will listen
26. en discuter - discuss it
27. Je lis le question at après ... I read 
28. vous regardez le photo et c'est qui dans le photo ...
29. demandez google
30. c'est quand fais ? - when is it ?
31. donner maximum information - give as much information as possible
32. s'appelle comment - is called what ?
33. vous présentez une personne et vous dites 
35. comme ça - like that
36. quelle personne ?

**_<u>Les Verbes</u>_**

1. première category ( les verbes réguliers )
2. deuxième category ( les verbes irréguliers )

**_première category :_**

1. er
2. ir
3. re

**Être ( verbe irrégulier )**

**_<u>Parler :</u>_**

Je parle
Tu parles
Il parle
Elle parle
Nous parlons
Vous parlez
Ils parlent
Elles parlent

**_<u>Chanter :</u>_**

Je chante
Tu chantes
Il chante
Elle chante
Nous chantons
Vous chantez
Ils chantent
Elles chantent

**_<u>Normandie :</u>_**

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kybju3gjc1qgja906fe5.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ksec8vz8zqicg7udogdm.png)

- Nord de France
- beaucoup de nature

**_<u>Quito - C'est où ?</u>_**

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/17m61f7vmll0kr6whpo5.png)

- Capitale de Ecuador
- l'Équateur (m) & équatorien(ne)


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/d0jduukyp2w003xegzk5.png)

- Il s’appelle Monsieur Narendra (le prénom) MODI (le NOM / le NOM de famille).
- Il est premier ministre d'Inde.
- Il est indien.
- Il parle anglaise, hindi et gujarati.
- Il habite à Delhi.
- Il est homme politique.	Il a 72 ans.
- Sa date d'anniversaire est le 17 septembre.
- Il est célibataire.
- Il vient de Gujarat. (venir de)
- Merci


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9foyhnpr31lu7hmr0if5.png)

- Il est célibataire.
- Il est président(e) et scientifique .
- date d'anniversaire - 15 Octobre
- Il parle Tamoul , Englais et Hindi.
- Il vient de Tamil Nadu.
- aujourd'hui , c'est spécial(e) en Inde.( 14th juillet ) -  Chandrayan 3.
- Il est connu " Missile Man de L'Inde "

**_<u>Important Points :</u>_**

les rubriques - Tabs / topics
émission - Program
Two Weeks of Coding
Tags: julialang, devjournal, gsoc
Published: 2023-07-16 18:55:09

Welcome, coding aficionados! It's time for a recap of the first two weeks of my GSoC journey with Julia. Now, don't get too excited, as the start of the project is often a slow one. These initial weeks were all about getting familiar with the project, diving into the codebase, and understanding the existing search functionality. So, grab your caffeinated beverage of choice, sit back, and let's embark on this slightly leisurely adventure!

![Beverage Sip](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/p2a9futekyzpqmh6j40v.gif)

During the initial weeks of GSOC, I mainly engaged in insightful discussions with my mentor, which proved to be incredibly valuable. These conversations allowed us to align our expectations for the new UI and dive deep into the intricacies of the search functionality. Together, we examined the existing flaws and brainstormed approaches to address them.

As we delved deeper into the search functionality, we realized the need for a smoother developer experience and a more maintained solution. It led us to explore the possibility of shifting from Lunr.js to MiniSearch.js. This shift to MiniSearch.js held the promise of streamlining our development workflow and reducing the overall maintenance burden. With its efficient indexing algorithms and smaller index sizes, we could deliver a snappier search experience to our users.


![Snap](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vh4sfn94m3vnu3huy9fw.gif)

These first two weeks of GSoC in the world of Julia might have been a bit slow and steady, but that doesn't mean we weren't making progress. From discussions about expectations and flaws to proposing new ideas for a search engine, we are setting the stage for exciting improvements. So, let's raise our virtual glasses to the future, where a revamped search UI will bring joy and efficiency. Stay tuned for more thrilling developments as this GSOC adventure unfolds!

![Prosit](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2zu0tbu1xjiky6r1yql0.png)

---

If you're interested in following my progress or connecting with me, please feel free to reach out!

Github: [Hetarth02](https://github.com/Hetarth02)
LinkedIn: [Hetarth Shah](https://www.linkedin.com/in/hetarth-shah-1ab392220)
Website: [Portfolio](https://hetarth02.github.io/)

Thank you for joining me on this journey, and look forward to more updates as I continue to contribute to the Julia community during GSoC.
What is an OpenAPI?
Tags: openapi, api, softwareengineering, networking
Published: 2023-07-16 18:45:37

Hey developers! 🖐️ Are you ready to take your API development to the next level? Today, I want to introduce you to a game-changing specification that is transforming the way we communicate between software applications: OpenAPI. 🚀

OpenAPI is redefining the API development landscape, enabling seamless communication, smoother interactions, and limitless integration possibilities. Want to dive deeper into this remarkable specification and uncover its pivotal role in API development? Check out our latest blog post for an in-depth exploration.

https://konfigthis.com/blog/openapi

Don't miss out on this captivating journey! Equip yourself with knowledge and stay ahead in the fast-paced world of software development. Let's unlock the power of OpenAPI together! 💪