teleren/devto huggingface dataset:
openGauss Parallel Page-based Redo For Ustore
Published: 2023-04-18 02:31:08

## Availability
This feature is available since openGauss 2.1.0.

## Introduction
Optimized Ustore inplace update WAL write and improved the degree of parallelism for Ustore DML operation replay.

## Benefits
The WAL space used by the update operation is reduced, and the degree of parallelism for Ustore DML operation replay is improved.

## Description
Prefixes and suffixes are used to reduce the write times of WAL update. Replay threads are classified to solve the problem that most Ustore DML WALs are replayed on multiple pages. In addition, Ustore data pages are replayed based on blkno.

## Enhancements
None.

## Constraints
None.

Dependencies
This feature depends on the Ustore engine.
openGauss Xlog no Lock Flush
Published: 2023-04-18 02:30:08

## Availability
This feature is available since openGauss 2.0.0.

## Introduction
Canceled the WalInsertLock contention and dedicated WalWriter disk write thread.

## Benefits
The system performance is further improved on the basis that the original Xlog functions remain unchanged.

## Description
This feature optimizes the WalInsertLock mechanism by using log sequence numbers (LSNs) and log record counts (LRCs) to record the copy progress of each backend. The backend can directly copy logs to the WalBuffer without contending for the WalInsertLock. In addition, a dedicated WALWriter thread is used to write logs, and the backend thread does not need to ensure the Xlog flushing.

## Enhancements
None.

## Constraints
None.

## Dependencies
None.
openGauss SMP for Parallel Execution
Published: 2023-04-18 02:26:15

## Availability
This feature is available since openGauss 1.1.0.

## Introduction
The Symmetric Multi-Processing (SMP) technology of openGauss uses the multi-core CPU architecture of a computer to implement multi-thread parallel computing, fully using CPU resources to improve query performance.

## Benefits
Fully utilizes the system multi-core capability to improve requery performance.

## Description
In complex query scenarios, a single query takes long time and the system concurrency is low. Therefore, the SMP technology is used to implement operator-level parallel execution, which effectively reduces the query time and improves the query performance and resource utilization. The overall implementation of the SMP technology is as follows: For query operators that can be executed in parallel, data is sliced, multiple working threads are started for computation, and then the results are summarized and returned to the frontend. The data interaction operator Stream is added to the SMP architecture to implement data interaction between multiple working threads, ensuring the correctness and integrity of the query.

## Enhancements
None.

## Constraints
Index scanning cannot be executed in parallel.
MergeJoin cannot be executed in parallel.
WindowAgg order by cannot be executed in parallel.
The cursor cannot be executed in parallel.
Queries in stored procedures and functions cannot be executed in parallel.
Subplans and initplans cannot be queried in parallel, and operators that contain subqueries cannot be executed in parallel, either.
Query statements that contain the median operation cannot be executed in parallel.
Queries with global temporary tables cannot be executed in parallel.
Updating materialized views cannot be executed in parallel.
Dependencies
None.
openGauss High Concurrency of Thread Pools
Published: 2023-04-18 02:24:29

## Availability
This feature is available since openGauss 1.0.0.

## Introduction
The thread pooling technology is used to support stable running of databases at high concurrency.

## Benefits
The overall system throughput is stable in case of a large number of concurrent requests.

## Description
The overall design idea of the thread pool technology is to pool thread resources and reuse them among different connections. After the system is started, a fixed number of working threads are started based on the current number of cores or user configuration. A working thread serves one or more connection sessions. In this way, the session and thread are decoupled. The number of worker threads is fixed. Therefore, frequent thread switchover does not occur in case of high concurrency. The database layer schedules and manages sessions.

## Enhancements
This feature is available since openGauss 1.0.0.

In openGauss 1.1.0, thread pools can be dynamically scaled in or out.

## Constraints
None

## Dependencies
None
openGauss Kunpeng NUMA Architecture Optimization
Published: 2023-04-18 02:20:30

## Availability
This feature is available since openGauss 1.0.0.

## Introduction
Kunpeng NUMA architecture optimization mainly focuses on Kunpeng processor architecture features and ARMv8 instruction set, and optimizes the system from multiple layers, including OS, software architecture, lock concurrency, logs, atomic operations, and cache access. This greatly improves the openGauss performance on the Kunpeng platform.

## Benefits
Transactions per minute (TPM) is a key performance indicator of the database competitiveness. Under the same hardware costs, a higher database performance means the database can process more services, thereby reducing the usage cost of customers.

## Description
openGauss optimizes the Kunpeng NUMA architecture based on the architecture characteristics. This reduces cross-core memory access latency and maximizes multi-core Kunpeng computing capabilities. The key technologies include redo log batch insertion, NUMA distribution of hotspot data, and CLog partitions, greatly improving the TP system performance.
Based on the ARMv8.1 architecture used by the Kunpeng chip, openGauss uses the LSE instruction set to implement efficient atomic operations, effectively improving the CPU usage, multi-thread synchronization performance, and Xlog write performance.
Based on the wider L3 cache line provided by the Kunpeng chip, openGauss optimizes hotspot data access, effectively improving the cache access hit ratio, reducing the cache consistency maintenance overhead, and greatly improving the overall data access performance of the system.
Kunpeng 920, 2P server (64 cores x 2, memory: 768 GB), 10 GE network, I/O: 4 NVMe PCIe SSDs, TPC-C: 1000 warehouses, performance: 1,500,000 tpmC.

## Enhancements
Batch redo log insertion and CLog partition are supported, improving the database performance on the Kunpeng platform.
Efficient atomic operations using the LSE instruction set are supported, improving multi-thread synchronization performance.
Constraints
None

## Dependencies
None
openGauss SQL by pass
Published: 2023-04-18 02:16:31

## Availability
This feature is available since openGauss 1.0.0.

## Introduction
Query performance is improved by customizing an execution scheme for typical queries in the TP scenario.

## Benefits
The TP query performance is improved.

## Description
In a typical OLTP scenario, simple queries account for a large proportion. This type of queries involves only single tables and simple expressions. To accelerate such query, the SQL bypass framework is proposed. After simple mode judgment is performed on such query at the parse layer, the query enters a special execution path and skips the classic execution framework, including operator initialization and execution, expression, and projection. Instead, it directly rewrites a set of simple execution paths and directly invokes storage interfaces, greatly accelerating the execution of simple queries.

## Enhancements
None

## Constraints
None

Dependencies
None
openGauss Hybrid Row-Column Store
Published: 2023-04-18 01:52:46

## Availability
This feature is available since openGauss 1.0.0.

## Introduction
openGauss supports both row-store and column-store models. Choose a row-store or column-store table as needed.

Column-store is recommended if a table contains many columns (called a wide table) but its query involves only a few columns. Figure 1 shows the column-store model. Row store is recommended if a table contains only a few columns and a query involves most of the fields.

Figure 1 Column-store

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/68x3dm52u5ihyppjyvrb.png)




## Benefits
In a wide table containing a huge amount of data, a query usually only includes certain columns. In this case, the query performance of the row-store engine is poor. For example, a single table containing the data of a meteorological agency has 200 to 800 columns. Among these columns, only 10 are frequently accessed. In this case, a vectorized execution and column-store engine can significantly improve performance by saving storage space.

## Description
Tables are categorized into row-store and column-store tables. Each storage model applies to specific scenarios. Select an appropriate model when creating a table.

## Row-store table

Row-store tables are created by default. Data is stored by row. Row-store supports adding, deleting, modifying, and querying data of a complete row. Therefore, this storage model applies to scenarios where data needs to be updated frequently.

## Column-store table

Data is stored by column. The I/O of data query in a single column is small, and column-store tables occupy less storage space than row-store tables. This storage model applies to scenarios where data is inserted in batches, less updated, and queried for statistical analysis. The performance of single point query and single record insertion in a column-store table is poor.

Selecting a storage model

Update frequency

If data is frequently updated, use a row-store table.

Data insertion frequency

If a small amount of data is frequently inserted each time, use a row-store table. If a large amount of data is inserted at a time, use a column-store table.

Number of columns

If a table is to contain many columns, use a column-store table.

Number of columns to be queried

If only a small number of columns (less than 50% of the total) is queried each time, use a column-store table.

Compression ratio

The compression ratio of a column-store table is higher than that of a row-store table. High compression ratio consumes more CPU resources.

## Enhancements
None

Constraints
None

Dependencies
None
openGauss Vectorized Engine
Published: 2023-04-18 01:50:49

## Availability
This feature is available since openGauss 1.0.0.

## Introduction
The vectorized execution engine, provided by openGauss, is usually used in OLAP data warehouse systems because analytical systems are usually data-intensive and access most data in a table in a sequential manner, perform calculation, and finally output a calculation result to an end user.

## Benefits
Batch calculation greatly improves the performance of complex query.

## Description
The traditional database query execution uses the tuple-based pipeline execution mode. In most time, the CPU is not used to actually process data, but to traverse the query operation tree. As a result, the effective utilization of the CPU is not high. This also results in low instruction cache performance and frequent jumps. Worse still, this approach does not take advantage of the new capabilities of the new hardware to speed up the execution of queries. In the execution engine, another solution is to change a tuple to a column at a time. This is also the basis of our vectorized execution engine.

The vectorized engine is bound to the column-store technology, because data of each column is stored together, and it may be considered that the data is stored in an array manner. Based on such a feature, when a same operation needs to be performed on the column data, calculation of each value of the data block may be efficiently completed by using a cycle.

The advantages of the vectorized execution engine are as follows:

This reduces inter-node scheduling and improves CPU usage.
Because the same type of data is put together, it is easier to leverage the new optimization features of hardware and compilation.
Enhancements
None

Constraints
None

Dependencies
It depends on column store.
openGauss LLVM
Published: 2023-04-18 01:49:36

## Availability
This feature is available since openGauss 1.0.0.

## Introduction
openGauss provides the Low Level Virtual Machine (LLVM) technology to query dynamic compilation execution.

## Benefits
The requery performance is greatly improved by dynamically building and executing queries.

## Description
Based on the query execution plan tree, with the library functions provided by the LLVM, openGauss moves the process of determining the actual execution path from the executor phase to the execution initialization phase. In this way, problems such as function calling, logic condition branch determination, and a large amount of data reading that are related to the original query execution are avoided, to improve the query performance.

## Enhancements
None

## Constraints
None

Dependencies
It depends on the LLVM open-source component. Currently, the open-source version 10.0.0 is used.
openGauss CBO Optimizer
Published: 2023-04-18 01:47:55

## Availability
This feature is available since openGauss 1.0.0.

## Introduction
The openGauss optimizer is cost-based optimization (CBO).

## Benefits
The openGauss CBO optimizer can select the most efficient execution plan among multiple plans based on the cost to meet customer service requirements to the maximum extent.

## Description
By using CBO, the database calculates the number of tuples and the execution cost for each step under each execution plan based on the number of table tuples, column width, null record ratio, and characteristic values, such as distinct, MCV, and HB values, and certain cost calculation methods. The database then selects the execution plan that takes the lowest cost for the overall execution or for the return of the first tuple.

## Enhancements
None

## Constraints
None

Dependencies
None
openGauss Basic Features
Published: 2023-04-18 01:46:18

Context
openGauss is a standalone database. It has the basic features of relational databases as well as enhanced features.

Features
Standard SQLs

Supports SQL92, SQL99, SQL2003, and SQL2011 standards, GBK, GB18030, UTF-8, SQL ASCII, and Latin-1 character sets, SQL standard functions and analytic functions, and stored procedures.

Database storage management

Supports tablespaces where different tables can be stored in different locations.

Primary/standby deployment

Supports the ACID properties, single-node fault recoveries, primary/standby data synchronization, and primary/standby switchover.

APIs

Supports standard JDBC 4.0 and ODBC 3.5.

Management tools

Provides installation and deployment tools, instance start and stop tools, and backup and restoration tools.

Security management

Supports SSL network connections, user permission management, password management, security auditing, and other functions, to ensure data security at the management, application, system, and network layers.
openGauss Operating Environment
Published: 2023-04-18 01:44:44

## Hardware
openGauss supports:

Arm server and x86_64-based universal PC server
SATA, SAS, and SSD local storage
1-gigabit and 10-gigabit Ethernet

**Supported OSs**
Arm:

openEuler 20.03LTS (recommended)
openEuler 22.03LTS
Kirin V10
Asianux 7.5
x86:

openEuler 20.03LTS
openEuler 22.03LTS
CentOS 7.6
Asianux 7.6
openGauss Product Features
Published: 2023-04-18 01:43:07

openGauss features high performance, availability, security, and maintainability.

## High performance

Achieves second-level response to queries of tens of billions of data through key technologies, such as the column-store, vectorized executor, and fusion engine.

## High availability (HA)

Intra-city cross-AZ DR ensures zero data loss and minute-level recovery.

## High security

Supports security features such as access control, encryption authentication, database audit, and dynamic data masking to provide comprehensive end-to-end data security protection.

## High reliability

TIMECAPSULE and RECYCLEBIN
Generally, it is difficult to recover table data that is updated, deleted, truncated, or dropped by mistake. In some cases, you can restore data to the time point before the mistaken operation, which is called point-in-time recovery (PITR). However, the restoration may make the entire database unavailable, and some table operations that users do not want to undo will also be undone. openGauss supports TIMECAPSULE and RECYCLEBIN. You can perform a flashback query or flash back a table based on timestamps to restore the data that is deleted or updated by mistake. By flashing back truncated or dropped table objects from RECYCLEBIN, you can restore table data to the state before the mistaken operation, which greatly improves the reliability of user data.

Good maintainability

Supports multiple maintenance methods, such as workload analysis report (WDR), slow SQL diagnosis, and session diagnosis, to accurately and quickly locate faults. It also provides the AI4DB capability and uses AI algorithms to implement database self-tuning, self-monitoring, and self-diagnosis.
openGauss System Architecture
Published: 2023-04-18 01:40:42

openGauss is a standalone database where data is stored on a single physical node and data access tasks are pushed to service nodes. In this way, high concurrency of servers enables quick data processing. In addition, data can be copied to the standby server through log replication, ensuring high reliability and scalability.

Software Architecture
openGauss is a standalone database and can be deployed in primary/standby mode.

openGauss shows the logical components of Figure 1.

Figure 1 openGauss logical components

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2r013rtp6x4drofso5sp.png)

## Architecture description

**OM**

Operation Manager (OM) It provides management interfaces and tools for routine maintenance and configuration management of the database.

**CM**

Acronym for cluster manager, a database management module. It manages and monitors the running status of functional units and physical resources in a database system, ensuring stable running of the entire system.

**Client driver**

Client driver receives access requests from the application layer and returns execution results. It communicates with openGauss instances, sends application SQL commands, and receives openGauss execution results.

**openGauss (primary/standby)**

openGauss primary/standby DN stores service data, executes data query tasks, and returns execution results.

openGauss supports one primary and multiple standbys. You are advised to deploy them on different physical nodes.

**Storage**

Functions as the server's local storage resources to store data permanently.
Building a Better Platform: Contractr Progress Report
Tags: webdev, javascript, programming, react
Published: 2023-04-18 01:11:38

Day 77 of #100DaysOfCode

Hello everyone,

It’s time for another update on my progress with the Contractr project! For those who are new to the project, Contractr is a web application that connects freelancers with potential clients. My goal with this project is to build a platform that makes it easier for people to find work or hire freelancers.

Today, I continued to work on the logic side of the project. I spent some time debugging and fixing issues, ensuring that everything is working as intended. While it wasn’t the most exciting day of coding, it’s an important part of the process, and it feels good to have everything running smoothly.

Moving forward, my focus will be on the chat feature. This will allow freelancers and clients to communicate with each other more easily and efficiently, making it simpler to discuss project details and requirements. I plan to work on storing conversations and other chat-related data starting tomorrow.

In addition, I’m considering implementing a search feature that will allow users to find freelancers or job postings based on specific criteria. This would be a valuable addition to the platform and make it easier for users to find the right fit for their needs.

Overall, I’m pleased with the progress I’ve made on the Contractr project so far. While there’s still plenty of work to be done, I’m excited to keep building and improving the platform. Thank you for your continued interest and support, and stay tuned for more updates soon.
Week 13: User deletion feature
Tags: opensource
Published: 2023-04-18 00:21:37

## Overview

As we wrap up week 13, we are inching closer to the launch of StartChart (aka MyCustomDomain) 1.0. Despite a busy week with school submissions, I managed to implement a crucial feature: user deletion, available exclusively for admins.

## Problem Statement

Deleting a user in a system often requires cascading deletion of all or most data associated with them. StartChart is no exception; our users have DNS records that need to be deleted as well. However, we decided not to delete SSL certificates for security reasons or other considerations.

## Implementation

Surprisingly, I didn't need to write any cascading deletion functionality myself. Prisma handles it automatically by deleting all entities referencing the deleted user, like DNS records in a many-to-one relationship. This revelation was quite impressive for me.

User deletion boiled down to two lines of code:

1. Deleting the user from the database (it cascades to DNS records)
2. Setting the reconciler to reconcile the database with Route53

Here's the code snippet:

```typescript
export async function deleteUser(username: User['username']) {
  await deleteUserByUsername(username);
  return setIsReconciliationNeeded(true);
}
``` 

The bigger issue was to integrate it in existing admin page. There were 2 type of actions before I added user deletion: 
- Searching users
- Impersonating users

With the addition of the third action, I refactored the code and assigned an `intent` to each action, which I believe is the recommended way to handle multiple actions in Remix. I also updated the zod schema and made some fields optional for reuse across different actions.

## Review
After I was done, created the PR and few iterations of reviews with minor suggestions it ended up being merged. 

https://github.com/DevelopingSpace/starchart/pull/607

## Afterthoughts
Last week was tough as I had to catch up with classes and submit various projects. Fortunately, I managed to do so successfully. It was also an Orthodox Easter and even tho I am consider myself as atheist, it was a very great time spent with my family that I really appreciated. I also signed up for the olympic distance triathlon that gonna be in the late July, and I feel like this is gonna be amazing (fun fact: I newer swam in pool for more than 50 meters and don't have a bike yet). Right now I just gotta do a final stretch and school is gonna be done, I am a one week closer to lifestyle I want to have. 

Oh, and also the last thing, I tried cold-showering for one minute in the morning (after like 10 mins of the warm shower as I usually have + going out for like 10 minutes right after that = feeling amazing and energetic throughout the day. Good luck with finals, everyone! Peace.
No Fee DAST Security Scans On Demand
Tags: security, testing, devops, coding
Published: 2023-04-18 00:06:45


End difficult, prolonged, resource-hungry #appsec testing with secure and thorough code inspection on-demand at(https://www.true-inspect.com.)  

**FREE** Standard (DAST)security scans with sign-up. 
No credit card, contract or minimum is required. 

Learn more:

**Clip
(https://video.wixstatic.com/video/edac60_5de94cd96893452c942e46245faaef83/1080p/mp4/file.mp4)

**PDF
(https://edac60cf-4062-4d32-80df-b179f00aeb91.usrfiles.com/ugd/edac60_4de829e6c9094749be74e81e38d9c61f.pdf)