Primary/Standby Shared Storage
Tags: opengauss
Published: 2023-03-10 03:57:02

**Introduction**

Shared storage enables the primary and standby nodes to share the same storage. In traditional HA deployment mode, the storage capacity is doubled compared with that of a single node. This feature provides a new HA deployment mode for the primary/standby shared storage, meeting the requirements for reducing the storage capacity and costs. In addition, OCK RDMA can be used to improve the real-time read consistency on standby nodes.

**Architecture**

The following figure shows the overall architecture of the primary/standby shared storage.

Figure 1 Primary/Standby shared storage architecture

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9ss8ennqtamjtam9z03c.png)

**·** UltraPath has been installed on the disk array, and the disk array is available.

**·** Distributed Storage Service (DSS)

The DSS process is an independent process that directly manages raw devices of disk arrays and provides capabilities similar to distributed file systems for external systems. The shared memory and client API dynamic library provide the database with the capabilities of creating files, deleting files, expanding and shrinking files, and reading and writing files.

**·** Distributed Memory Service (DMS)

DMS is a dynamic library integrated in the database. It transmits page content through the TCP/RDMA network, integrates the primary and standby memories, and provides the memory pooling capability to implement real-time read consistency on standby nodes.

**·** Page exchange between the primary and standby nodes is accelerated by RDMA, depending on the CX5 NIC and OCK RDMA dynamic library.

**Features**
**·** The primary and standby nodes share one copy of data, significantly reducing the storage capacity in traditional HA deployment mode.

**·** Log replication is not required between the primary and standby nodes. Instead, page exchange is implemented between the primary and standby nodes and real-time read consistency is supported on standby nodes.

**·** By default, the primary and standby nodes exchange pages in real time through the TCP network. To reduce the page exchange delay, the OCK RDMA dynamic library is introduced to improve the real-time consistency performance of the standby node.

**Applicable Scenarios and Restrictions**

**·** The storage device is a disk array. The LUN of the disk array must support the SCSI3 PR protocol, including PERSISTENT RESERVE OUT (PR OUT), PERSISTENT RESERVE IN (PR IN), and INQUIRY, to implement cluster IO FENCE. In addition, the COMPARE AND WRITE (CAW) protocol of SCSI3 must be supported to implement shared disk locks. For example, Dorado 5000 V3 disk array.

**·** A maximum of eight standby nodes can be deployed in openGauss.

**·** The primary/standby shared storage depends on functions similar to the distributed file system to implement the real-time read consistency on standby nodes. Therefore, the less the file metadata changes, the better. To ensure performance, this feature supports only segment-page tables.

**·** The primary and standby nodes must be deployed on the same disk array. DR deployment and hybrid deployment of the primary and standby nodes are not supported. For example, the primary and standby nodes are deployed on different disk arrays.

**·** Page exchange between the primary and standby nodes is accelerated by RDMA, depending on the CX5 NIC and OCK RDMA dynamic library.
**·** Currently, standby node rebuild, node replacement, and node restoration are not supported.
**·** Upgrade from the traditional HA deployment to the primary/standby shared storage deployment is not supported.

**Manual Installation Example**

Compared with traditional gs_initdb database creation, directories are classifies into three types during shared storage database creation: exclusively used and not shared by each instance, exclusively used and shared by each instance, and shared by all instances. The directories to be shared must be stored on the disk array, and the directories that are not shared must be stored on the local disk. In addition, to create a database on the standby node, you only need to create a directory that belongs to the standby node. You do not need to create a directory structure shared by all instances. GUC parameters are added to the primary/standby shared storage, and the storage mode of system catalogs is switched from page mode to segment-page mode.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nn0jr84yvq6l83u59tpe.png)

**·** Prerequisites
**-** The code of the project has been compiled. For details about the compilation, see Software Compilation and Installation.
**-** The LUN of the disk array has been mounted to the host, the UltraPath software has been installed, and the disk array is available.

**·** Procedure

1. Create symbolic links for drive letters of the disk array LUN and assign the corresponding permissions to the user. (Assume that the drive letters of the LUN are /dev/sde and /dev/sdf.)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bbr1wbpsbsekh2b9vcd4.png)

2. Assign the RAWIO permission of the disk array to the required executable files.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/re27brsqt29dedv7bku1.png)

**perctrl**: executable tool used to assign read and write permissions to DSS-related tools and processes.

3. Create a DSS server process and a configuration file required for creating a shared storage database.

Test directory (for example, /**data/test**)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kapf2qla6osnandmt6zo.png)

The content of the dss_init.ini file is as follows:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/n28i4a8v0205v4nnfch3.png)

The parameters in the preceding configuration are described as follows:
3.1. **INST_ID** indicates the instance ID. The value ranges from 0 to 63. The value varies according to the primary and standby dssserver processes.
3.2. **_LOG_LEVEL** indicates the log level.
3.3. **_LOG_BACKUP_FILE_COUNT** indicates the maximum number of log files that can be retained.
3.4. **_LOG_MAX_FILE_SIZE** indicates the maximum size of a log file.
3.5. **LSNR_PATH** indicates the directory for storing the domain socket used for communication between the DSS client and server. Generally, it is set to the home directory of the DSS server process.
3.6. **STORAGE_MODE** indicates the storage device type of the DSS. Set this parameter to **RAID** for disk arrays.
3.7. **_SHM_KEY** indicates the shared memory key. Ensure that each DSS has a unique key.

The content of the dss_vg_conf.ini file is as follows:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/k0htjfqpgym73j2jqdb0.png)

The preceding information indicates that the contents of the **+data** directory are stored on the **/dev/tpcc_data** device and the contents of the **+log** directory are stored on the **/dev/tpcc_log** device. Check whether the root directory name contains the plus sign (+) to determine whether a file is in the file system or DSS. DSS is regarded as a distributed file system.

4. Use the DSS client tool (dsscmd) to initialize the volume group (VG) on the disk array (similar to initializing the file system on a raw disk).

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pa18lrtb1f2b8pvxgg43.png)

5. Use gs_initdb to create a shared storage database.

gs_initdb -D /data/ss_test/dn_primary --nodename=single_node -w Gauss_234 --vgname="+data,+log" --enable-dss --dms_url="0:127.0.0.1:1611,1:127.0.0.1:1711" -I 0 --socketpath="UDS:/data/ss_test/dss_home/.dss_unix_d_socket"

The following parameters are added:

5.1. –**vgname** specifies the VG where the shared storage database is created. The VG name is related to the configuration item in the **dss_vg_conf.ini file**. The VG name must exist in the configuration file and start with a plus sign (+).
5.2. –**enable_dss** indicates that the shared storage database created in DSS.
5.3. –**dms_url** “**0:127.0.0.1:1611,1:127.0.0.1:1711**” indicates that format is Instance_ID**:**IP address**:**Port number.
5.4. -**I**: specifies the instance ID of the current node. The value ranges from 0 to 63.
5.5. –**socketpath** specifies the location of the Unix domain socket used for the communication between the DSS client (DSS client dynamic library integrated into the executable file of the database) and the server.
6. After the shared storage database is successfully created, run the gs_ctl start command to start the database process.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ievwrvcqdoxcptkyaplh.png)

7. Repeat the preceding installation procedure on the standby node.

**OCK RDMA Usage Example**

The shared storage feature provides the real-time read consistency function on standby nodes. RDMA can be used to accelerate page exchange between the primary and standby nodes. This section describes how to enable the RDMA communication function in the shared storage.

**·** Prerequisites

The RDMA function depends on the CX5 NIC and OCK RPC dynamic library.

**·** Procedure

1. Contact openGauss support to obtain OCK RPC package.
2. Deploy the OCK RPC package.

2.1 Decompress the OCK RPC package.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/tprjh42pc8s0opdnrtto.png)

2.2 Modify the parameters in the ock_rpc_opengauss.sh deployment script.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zl56lmpd7mxlvdiuwr8j.png)

The parameters involved in the script are as follows:

2.2.1 USER_NAME: user name of the host where the DN is located.
2.2.2 HOST_IP: IP address of a cluster host, including the local host.
2.2.3 PACKAGES: name of the obtained OCK RPC binary package (matching the host OS architecture).
2.2.4 PATH_TO_SCP: copy path.
2.2.5 GAUSSHOME: The value must be the same as the environment variable GAUSSHOME configured on openGauss.
2.2.6 OCK_RPC_LIB_PATH: path of the OCK RPC binary file. The shared storage is used in dynamic loading mode. Ensure that the path is correct.

2.3 Run the deployment script ock_rpc_opengauss.sh.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lu462y4rsr7qj5bipn5a.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/b0d6wfox8c05i5ollw94.png)

2.4 Configure other environment variables.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kyo1cb263vvxo2ac4jjr.png)

Parameter description:

2.4.1 The **UCX_ZCOPY_THRESH** optimization parameter is used to reduce the memory application during RDMA transmission. Unit: byte. Default value: **32768**. (This parameter is optional. The default value is recommended.)

2.4.2 The **UCX_USE_MT_MUTEX** optimization parameter is used to ensure the data consistency lock during RDMA transmission. The options are **n** (atomic lock) and **y** (mutex). (This parameter is optional. You are advised to use the atomic lock because the performance is good.)

2.4.3 The **UCX_NET_DEVICES** configuration parameter is used to specify the NIC port. You can install the mlnx_ofed driver and run the **ibdev2netdev** command to view the current active RDMA ports. (You are advised to specify active ports.)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/49vozlvbu2cyj9og8d4u.png)

3. Perform steps 1 to 5 to gs_initdb to create a database.

4. Modify the **postgresql.conf** file.

Set **ss_interconnect_type** to **RDMA** and add the **ss_rdma_work_config** and **ss_ock_log_path** configuration items. For details about the parameter description and configuration, see Shared Storage Parameters.

Example:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1yxff9q4s3yl7cdkimnh.png)

5. Run the gs_ctl start command to start the database process.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kpni8jqf62ck34uup8p6.png)