teleren/devto huggingface dataset:
The 20th Day of the 100 Days of Code challenge
Published: 2023-06-29 21:05:45

Today, I made a working demo for my real-time transcription app that uses Whisper and has Speaker Diarization as well.

The problem I've been having for ages with transmitting valid audio I can save to a file from the client was quickly solved by taking inspiration from another project that transcribes audio in real-time, just using Google's API.

A project by Sahar Mor, saved me a lot of time by already having the general logic needed for this kind of application that multiple clients might be using.

A quick look at the backend outlined the process:

- Client identification by Session ID, allowing us to easily access and store its transcription data

- Separate transcription thread to keep the flow coming.

Here are the current challenges:

First challenge:
I don't wanna say anything about any library, but I feel like choosing to extract speaker embeddings with pyannote's model was a poor choice. Since the embeddings are the most important part (there's no need to diarizate audio recordings as Whisper provides segments when transcribing), and the segments can be very short, it's important to have an amazingly accurate library to extract them. Aiming to make use of a library that has a larger dataset and is more renowned for its accuracy.

Second challenge:
Transcription accuracy.
Whisper's transcription accuracy is as good as it gets, but that depends on the model. For some reason, I get execution times as long as 35 seconds for a few-second audio file when using the large model. However, a friend said he managed to get under 2 seconds when running the server in Google Colab. That's what I've got on the agenda for tomorrow.

I initialize the model when the server initializes, I have no idea why it takes so long when running on PyCharm. I think that I've done something wrong when calling it.

The current demo isn't very promising at first sight, but the fixes don't appear to be too demanding.

The accuracy of the Speaker Diarization will improve when the approach improves, and the same goes for the transcription accuracy.

That's it for today,
Happy coding everyone!

* Demo is available on Twitter
Next.js 13.4 App Router + Express Backend
Published: 2023-06-29 20:13:58

Hi all!  

I am having issues configuring Next.js – with the newly stable App Router, so v. 13.4.7 -- with an Express.js backend.  I’ve been looking for an example project on Github and Udemy, and haven’t been able to find one.

Can anyone point me to an example combining Next's new App Router with Express?

Thanks!
Johnson
Sharkio: The first contribution
Published: 2023-06-29 20:05:18

Hello world.

My name is Ido.
I am a senior software engineer. I am a programmer from a very young age. I love code and I love open source.

I wanted to share with you something that is very exciting!

Recently I created My first open source project called Sharkio. It is a developer tool that aims to help me and other Backend/API/Fullstack developers in their daily work.

I joined an awesome open source community. And very quickly developers showed interest in my project. And after a very short time they started running the project asking questions. And be more involved.

This is a dream come true for me. Sharing my vision with fellow developers. Them showing interest and seeing the potential. And them eventually contributing code.

If you want to learn more about sharkio. Please feel free to reach out to me on discord and on github:

[Sharkio Github](https://github.com/idodav/sharkio)
[Sharkio discord](https://discord.gg/KtM2WzPu)
How to Set Up Quay.io/Keycloak with Docker Compose for Local Development?
Tags: webdev, keycloak, docker, authentication
Published: 2023-06-29 19:53:08

Hey there! Are you looking for a powerful and easy-to-use Identity and Access Management (IAM) solution for your app? Look no further than `Quay.io/Keycloak` In this blog, I’ll show you how to set up and use `Quay.io/Keycloak` with docker-compose for local development. 

```
version: '3.8'

services:
  keycloak_database:
    container_name: keycloak_database
    hostname: keycloak_database
    image: postgres:latest
    environment:
      POSTGRES_DB: keycloak
      POSTGRES_USER: keycloak
      POSTGRES_PASSWORD: keycloak
    ports:
      - 5432:5432
    volumes:
      - "./keycloak-dump.sql:/docker-entrypoint-initdb.d/keycloak-dump.sql:ro"

  keycloak:
    container_name: keycloak
    hostname: keycloak
    image: quay.io/keycloak/keycloak:latest
    environment:
      KEYCLOAK_ADMIN: admin
      KEYCLOAK_ADMIN_PASSWORD: admin
      KC_DB: postgres
      KC_DB_URL_HOST: keycloak_database
      KC_DB_URL_PORT: 5432
      KC_DB_URL_DATABASE: keycloak
      KC_DB_USERNAME: keycloak
      KC_DB_PASSWORD: keycloak
    command: start-dev
    ports:
      - 8080:8080
    depends_on:
      - keycloak_database

```
`docker-compose up` will start a postgres container with prefilled data stored in `keycloak-dump.sql` file and you will be able to access keycloak instance in `localhost:8080`.

_Just a heads up, there are already plenty of articles out there about implementing Keycloak in a web server. To keep things simple, I’ve decided not to include that part here. But if you’re interested in learning more, feel free to let me know in the comments!_ 😊

<u>Helpful links</u>
1. [Keycloak](https://www.keycloak.org/)
2. [Keycloak-getting-started-docker](https://www.keycloak.org/getting-started/getting-started-docker)
Destructuring Arrays in JS
Tags: javascript, webdev, programming, beginners
Published: 2023-06-29 19:44:29

Destructuring arrays in JavaScript allows you to extract values from an array and assign them to variables in a concise and convenient way. It provides a shorthand syntax for assigning values to variables from specific positions within an array.

Here's an example of array destructuring in JavaScript:

```javascript
const myArray = [1, 2, 3, 4, 5];

// Destructuring assignment
const [first, second, , fourth] = myArray;

console.log(first);   // Output: 1
console.log(second);  // Output: 2
console.log(fourth);  // Output: 4
```

In the above example, we have an array `myArray` with five elements. By using destructuring assignment, we declare variables `first`, `second`, and `fourth` and assign them the corresponding values from the array. The third element of the array is skipped using an empty slot in the destructuring assignment.

Array destructuring also supports default values. If an element in the array is `undefined` or absent, the default value will be used. Here's an example:

```javascript
const myArray = [1, 2];

const [first = 0, second = 0, third = 0] = myArray;

console.log(first);   // Output: 1
console.log(second);  // Output: 2
console.log(third);   // Output: 0 (default value)
```

In the example above, the array `myArray` has only two elements. By providing default values (`0` in this case) for the variables `first`, `second`, and `third`, even if the corresponding elements are absent, the destructuring assignment will assign the default values.

You can also use the rest parameter syntax (`...`) to capture the remaining elements of an array. Here's an example:

```javascript
const myArray = [1, 2, 3, 4, 5];

const [first, ...rest] = myArray;

console.log(first);  // Output: 1
console.log(rest);   // Output: [2, 3, 4, 5]
```

In the example above, the variable `first` captures the first element of the array, and the `rest` variable uses the rest parameter syntax (`...`) to capture the remaining elements as an array.

Array destructuring can be particularly useful when working with functions that return arrays or when extracting values from complex data structures like objects containing arrays.

Remember that array destructuring relies on the position of elements within the array. If you only need to access specific properties of an object, you should use object destructuring instead.
Society is a DB
Tags: sql, python, programming
Published: 2023-06-29 19:38:36

As I was learning about databases in phase 3 at Flatiron, I started to notice similarities between the way databases (DB's) and society (particularly here in the U.S.) manages data. This really dawned on me when I learned that each table in a DB keeps track of each row/entry with a unique ID. Thinking about your driver's license or personal ID card, you might realize that the unique ID number on the card, which is in the state system, is simply a unique ID on an enormous table of data that contains you as an entry/row. This blew my mind when I thought of it this way, especially when I considered that digital DB's have only been around for around 20-30 years. Before computers, these huge DB's containing all of our identifying information was stored in filing cabinets. 
    
You might imagine what other attributes or columns might be on a given table for a U.S. state's DB. You have your unique ID which is just an integer. But then you also have Boolean data. Has a given entry (person) demonstrated through testing that they are capable of operating a vehicle safely and are over the age of 16? True or false? Has a given entry given consent to donate their organs to medical causes should they meet an untimely demise? True or false? And then our state's DB stays updated with other data like our weight, height (INT's), whether or not we can see well without glasses (Bool), and also a good old JPEG with an image of each entry from the chest up.
    
It was really interesting to consider how dependent we are on using numerical, text and binary data to organize systems in our societies. So much of our lives revolve around data. Whether it's your lunch number, the unique magnet in your credit card, your Flatiron student ID number or your Facebook account password, you can rest assured that you are an entry in a DB, somewhere.
Code Usability and my "Ease of Use" Levels
Tags: javascript, api, development, programming
Published: 2023-06-29 19:29:33

In my experience, the concept of "code usability" doesn't get enough attention. When you create a library or module, its usability can have a huge impact on the consumer's productivity. I identified what I find makes code easy to use and came up with these 3 "ease of use" levels.

_DISCLAIMER: These are just my ideas, not "official" by any means._

In the following examples, imagine we're using a geometry library, and we want some way to represent a pyramid shape in our code.

## Level 1: Easy to use
You can easily figure out how to use it by reading documentation or examples.

✔️ Simple
✔️ Well-documented

**Example:**
![level 1 example](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hvf8vb0gbzy8k0g28ndy.png)

---

## Level 2: Intuitive to use
You can guess how to use it _without_ reading documentation, usually based on function, class, or variable names.

✔️ Good naming

**Example:**
![level 2 example](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vnl2gjgr6cb657nocf51.png)

---

## Level 3: "Uses itself"
You can use it without even thinking about it, usually with code assist (intellisense). It also prevents you from misusing it by mistake.

✔️ Facilitates code assist
✔️ Thorough error handling

**Example:**
![level 3 example](https://i.imgur.com/Dlpgkpy.gif)

---

# Takeaway

Sadly, level 3 isn't always a given. Try using a library with lackluster documentation, or accidentally forget a required argument, and it could set you back hours. So, how can _you_ write usable code? Well in JavaScript, you can get pretty far with good [JSDoc](https://jsdoc.app/) comments.

**Example:**
```javascript
/**
 * Returns an object representing a pyramid with the given dimensions.
 * @param {number} length
 * @param {number} width
 * @param {number} height
 * @returns {{length: number, width: number, height: number, volume: () => number}}
 * @example
 * const p = createPyramid(2, 3, 4)
 * const v = p.volume()
 */
export function createPyramid(length, width, height) {
  return {
    length,
    width,
    height,
    volume() {
      return (length * width * height) / 3;
    },
  };
}
```

The rest comes down to your design decisions. Try putting yourself in the shoes of the consumer, and test if your code reaches level 3. You may be surprised how much you can help the future consumer... even if it ends up being YOU.

💬 Feedback/criticism are welcome below 👇
Vacuum Processing in PostgresSQL
Tags: database, sql, postgressql
Published: 2023-06-29 19:26:24

Vacuum processing is one of the core mechanism in PostgresSQL. It basically involves the removal of dead tuples in the database. In order to remove the dead tuples, it involves two modes

- Concurrent VACUUM
- Full VACUUM

##Concurrent VACUUM:
Concurrent VACUUM also known as VACUUM. It removes dead tuple for each page of the table and other transactions can read the data while the process is still running. It involves three blocks:

####First Block: 
This block performs the freeze processing and removes index tuples that points to dead tuples.
####Second Block:
This block removes the dead tuples and update both Free space map (FSM) and Visibility map (VM).
####Third Block:
This block performs the cleanup after the deletion of indexes and update both the statistics and the system catalog.

####Post-processing:
When the vacuum processing is complete, after updating both statistics and system catalogs related to vacuum processing, it removes unnecessary parts of the clog if possible.

##Visiblity Map:
Vacuum processing is quite costly so visibility map was introduced. Each table has a visibility map that holds the visibility of the page in table file and determines the dead tuples.

##Freeze Processing:
Freeze Processing has two modes

- **Lazy Mode:** When freeze processing scans only pages that contain dead tuples using the visibility map of the target tables.

- **Eager Mode:** It scans all pages to inspect all tuples in tables and update relevant system catalogs and removes unnecessary files.

##Full VACUUM:
Concurrent vacuum is essential but it is not sufficient so here comes the full vacuum. It not only removes the dead tuples but also reduced the table size. But two things should be considered here:

- Nobody can access the table when Full VACUUM is processing.
- At most twice the disk space is used temporarily.

##References:
[https://www.interdb.jp/pg/pgsql06.html](url)
Internals of PostgreSQL - Chapter 5
Published: 2023-06-29 19:26:15

Concurrency control is a process that maintains atomicity and isolation, properties of ACID, of which there are three techniques. In Multi-version Concurrency Control, each write operation creates a new data item while retaining the old version which is in contrast to Strict Two Phase Locking which must block readers when a writer writes an item because the writer acquires an exclusive lock for the item. PostgreSQL used Snapshot Isolation which is a variation of MVCC which uses rollback segments. When reading items, PostgreSQL selects the appropriate version of an item in response to an individual transaction by applying visibility check rules.
Whenever a transaction begins, a unique identifier, referred to as a transaction id (txid) which is a 32-bit unsigned integer, approximately 4.2 billion. 
Heap tuples in table pages are classified as a usual data tuple and a TOAST tuple. This section describes only the usual tuple.
While the HeapTupleHeaderData structure contains seven fields, four fields are required in the subsequent sections which are t_xmin, t_xmax, t_cid and t_ctid
FSM is used to insert and update tuples. 
For insertion, a new tuple is inserted directly into a page of the target table. For this the header fields would be as follows:
t_xmin 	= 99
t_xmax =0
 t_cid =0
t_ctid =(0,1)

For deletion, target tuple is deleted logically and in header fields, t_xmax is set to 111
In the update operation, PostgreSQL logically deletes the latest tuple and inserts a new one
Tuple_1:
t_xmax is set to 100.
t_ctid is rewritten from (0, 1) to (0, 2).
Tuple_2:
t_xmin is set to 100.
t_xmax is set to 0.
t_cid is set to 0.
t_ctid is set to (0,2)


When inserting a heap or an index tuple, PostgreSQL uses the FSM of the corresponding table or index to select the page which can be inserted it. PostgreSQL  , and is used throughout transaction processing. PostgreSQL defines four transaction states, i.e. IN_PROGRESS, COMMITTED, ABORTED, and SUB_COMMITTED. A transaction snapshot is a dataset that stored information about whether all transactions are active, at a certain point in time for an individual transaction. Here an active transaction means it is in progress or has not yet started. Visibility check rules are a set of rules used to determine whether each tuple is visible or invisible using both the t_xmin and t_xmax of the tuple, the clog, and the obtained transaction snapshot.
Typescript Flatten Object Type
Tags: webdev, javascript, typescript, tutorial
Published: 2023-06-29 18:54:03

```ts
type ToPaths<T, P extends string = ''> = T extends Record<number, unknown>
    ? {
        [K in keyof T]: ToPaths<T[K], `${P}${K & string}.`>
    }[keyof T]
    : { path: P extends `${infer P}.` ? P : never; type: T }

type FromPaths<T extends { path: string; type: unknown }> = {
    [P in T['path']]: Extract<T, { path: P }>['type']
}

type A = FromPaths<ToPaths<{
//  ^?
    a: 1,
    b: {
        c: 2,
        d: Record<string, {
            e: 3,
            f: 4,
        }>,
        g: 5,
    },
    h: {
        i: {
            j: 6,
        },
    },
}>>


```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gvnh7br5rcz9fofzim3o.png)
[playground](https://www.typescriptlang.org/play?#code/C4TwDgpgBAKg9gBQIbABYGcA8MA0UFQQAewEAdgCbpTrABOAlmQOZQC8UA5JwHzuyES5KlABKEAMZw6FTGQCuAWwBGEOnnlkA1mTgB3MjwBQUU1AD8UAN4mzdgNoBpKEyhaIIOADNYAXQBcsIgoGNhOvngABgAkVggAvrHOAGQ09EzM8QB0kcZ28fbunj4wvramgVZQYCGBBMSklNQxVkxeavjZkRb4UIFkEABuagDcUKCQgTBQ8UZGE9AAYnRwishoWNMNwtRVNWiBtIwsYwuBmjr6ZDN8HDYOBK4w9pz7qJy+AVAAoiR0SBJgNg8Htar14jwXgsPkZZvNwNAAIL8ZardaheDorA2AD0ONMAD1zOUoEhAgBGHAk5SVEl2CSBABMVLsdgogXEUhkmCOGRBdNZpgggQAzCzBayvIEACziwUQuWs5iBACsiviitQtIlLm1OrMACtAgA2RX5dVUiHGIxAA)
Concurrency Control in PostgresSQL
Published: 2023-06-29 18:16:15

Concurrency control is the process of maintaining atomicity and isolation when two transactions run concurrently in the database. 

There are three types of concurrency control techniques:

- **Multi-version Concurrency Control (MVCC)** is the technique that allows multiple transactions in database without blocking each other. PostgresSQL and some other RDMS uses the version of MVCC called as **Snapshot Isolation (SI)**.

- **Strict Two-Phase Locking (S2PL)** is the technique that uses locks while accessing the shared resources in database. It means that if one transactions is happening, the lock will prevent other transactions to access the shared resource.

- **Optimistic Concurrency Control** works by reducing the need of locks during transactions in database. It supposes that the conflicts are not often. As described by its name, OCC works optimistically and allows transactions to happen but if the conflict occurs,  OCC revert the transaction that causes the conflict.

###Transaction ID: 
The unique identifier (txid) assigned by the transaction manager to a transaction is known as transaction ID.

PostgresSQL has three types of txids:
**0** means invalid txid.
**1** means Bootstrap that is used in the initialization of database cluster.
**2** means Frozen txid that is used in the for MVCC.

###Commit Log:
Commit log holds the statuses of transactions. This log is allocated to the shared memory and is used throughout the transaction processing.
There are four transactions statuses in the commit log:

- `IN_PROGRESS` when transaction is in progress.

- `COMMITTED` when transaction completed successfully.

- `ABORTED` when transaction encounters some errors.

- `SUB_COMMITTED` when transaction goes under some custom implementations.

###Transaction Snapshot:
Transaction snapshot refers to the dataset that stores all the information about the transaction like whether it is active, at a certain time for an individual transaction.

###Serial Snapshot Isolation:
This concurrency method is used to make sure the high level of isolation. One transaction never gets the effect of other transactions in database and it can never read the data that is added by another transactions in the database but not committed successfully i.e. serialization.

##References:
[https://www.interdb.jp/pg/pgsql05.html](url)
Looking to find interesting people!
Tags: webdev, discuss, programming, community
Published: 2023-06-29 18:12:09

Hello everyone,

My name's Ibrahim, I'm a high school student with over 6 years of experience coding in all sorts of frameworks and languages. Only recently, I've decided to start building projects for clients, and I didn't really know where to start.

After doing research and coming up with ideas of my own, I've decided that spending my time building connections with other developers and partners can be one of the best things to do with my time. I want to meet new people not just for business purposes as well, but to expand my knowledge and expertise and potentially make some new friends in the industry.

Anyway, If you can, replying to this post would mean the world to me as I feel that it can help jumpstart a new view on the programming world.

(PS. if you want to see my software company, just visit https://www.nexieon.com for more information)
Did the Amazon Prime Team Get it Wrong? 🤔
Tags: microservices, softwareengineering, distributedsystems, watercooler
Published: 2023-06-29 18:08:41

Microservices fail when the engineers involved only focus on the how, and disregard the "what" and "why" parts. If there were tightly coupled microservices in the Amazon Prime Video architecture, no wonder it failed to be an adequate solution.

Microservices should be loosely coupled. Each microservice should be unaware, as far as possible, of the existence of other microservices. Each microservice should "think" it's the only service in the world.

They should have followed these principles instead:

{% embed https://youtu.be/E_XOIHeHkB4 %}
The Easiest Factory Pattern Explanation
Tags: java, designpatterns, coding, programming
Published: 2023-06-29 17:54:20

{% embed https://youtu.be/iTfa97vXCAo %} 

In software engineering, the Factory pattern is a creational pattern that creates objects without exposing the creation logic to the client and refers to newly created objects using a common interface. In other words, it encapsulates object creation and decouples it from the client code that uses the objects. This can improve code maintainability, flexibility, and reusability.

For example, imagine you have a program that needs to create different types of objects, such as shapes or vehicles. You could write a bunch of conditional statements or switch statements in your client code to determine which concrete class to instantiate. However, this approach can quickly become messy and hard to extend. Instead, you could use a factory method or a factory class that takes care of the object creation based on some input parameter or logic.

Some benefits of using the Factory pattern include:
👉 Encapsulating object creation and reducing coupling
👉 Providing a central point of control and configuration for object creation
👉 Promoting the open-closed principle by allowing new subclasses to be added without modifying existing code
👉 Enabling testing and mocking of the client code without actually instantiating real objects

So, if you want to improve the flexibility and maintainability of your software, consider using the Factory Design Pattern. It's a powerful tool in your design patterns toolbox. 🧰