AntDB Installation and Deployment Documentation - P6
Tags: database
Published: 2023-08-02 02:08:48

## ETCD+patroni for high availability
etcd is responsible for the storage of cluster status information, which is used to contact each node, and patroni is responsible for providing high availability service for the cluster, the collection of both provides failover high availability service for AntDB standalone cluster.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8udp8xkwvv03s3t0wozj.png)

**Note:** The following steps are all required to be executed on all hosts where the primary and backup are located. Therefore, the following steps need to be performed here on all 3 servers.
**Install python3**
1. Download the installation package, the recommended version is greater than or equal to 3.7, or get the installation package from AntDB operation and maintenance staff.
https://www.python.org/downloads/source/
To solve the problem that the '_ctypes' module of python cannot be found, you need to install this package additionally
#root user to execute it, or use antdb sudo to execute it

```
yum install -y libffi-devel
```

2. decompress it.

```
tar -zxvf Python-3.7.0.tgz
or
unzip Python-3.7.0.tgz
```
3, compile and install, here executed under the root user.

```
cd Python-3.7.0
./configure --prefix=/usr/local/
make
make install
```
4, create hyperlink

```
ln -s /usr/local/bin/pip3 /usr/bin/pip3
ln -s /usr/local/bin/python3 /bin/python3
```
5、Check python3 version
 

```
python3 -V
```

**Install psycopg2**
Install dependencies.
 

```
yum install -y postgresql-devel
```
Use the pip command to install psycopg2 or psycopg2-binary.

```
pip3 install psycopg2 
```
If the pip command fails to install, you can download the psycopg2 installation package and install it manually.

```
tar -xzvf psycopg2-2.9.3.tar.gz
cd psycopg2-2.9.3
python3 setup.py build
python3 setpy.py install
```
Check if the installation is successful

```
python3 -c "import psycopg2; print(psycopg2.__version__)"
```
Normal result: No error is reported and the version number is output.
**Install patroni**
Install using the pip command under the root user.

```
pip3 install patroni
```
If the installation package is available, you can also install it by following, --find-links is the file path.

```
tar -xzvf patroni-2.1.2.tar.gz
pip3 install --no-index --find-links=file:/data1/adbpuq/patroni_offline_pg13/patroni-2.1.2 patroni
pip3 install --no-index --find-links=file:/data1/adbpuq/patroni_offline_pg13/patroni-2.1.2 python-etcd
```
Check Patroni

```
which patroni
patroni -h
```
**Install etcd**
Executed by root user

```
yum -y install etcd
```
Or install via the installation package at

```
tar -xzvf etcd-v3.3.18-linux-amd64.tar.gz
cd etcd-v3.3.18-linux-amd64
cp etcd /usr/bin
cp etcdctl /usr/bin
```
Check etcd

```
which etcd
which etcdctl
```
**Build ETCD cluster
Configure etcd**
Execute under the root user
1. Create etcd configuration directory and data directory

```
mkdir /etc/etcd
mkdir -p /var/lib/etcd/data
```
2. Configure etcd
Write the configuration of etcd to the yaml file, for example, on the host with IP 10.20.16.227, the configuration is as follows.

```
cat > /etc/etcd/conf.yaml <<EOF
name: etcd-1
data-dir: /var/lib/etcd/data
listen-client-urls: http://10.20.16.227:12379,http://127.0.0.1:12379
advertise-client-urls: http://10.20.16.227:12379
listen-peer-urls: http://10.20.16.227:12380
initial-advertise-peer-urls: http://10.20.16.227:12380
initial-cluster: etcd-1=http://10.20.16.227:12380,etcd-2=http://10.20.16.228:12380,etcd-3=http://10.20.16.214:12380
initial-cluster-token: etcd-cluster-token
initial-cluster-state: new
EOF
```
Parameter descriptions.

- name: Each node is different, and each member must have a unique name.

- data-dir: The directory of the etcd data file created above, where the data is saved when the service is running.

- listen-client-urls: listen address, the address of the external service.

- advertise-client-urls: the listening address used to notify other ETCD nodes that clients are accessing this node.

- listen-peer-urls: the listening address for this node to exchange data with other nodes (election, data synchronization), used to listen to other members.

- initial-advertise-peer-urls: address to notify other nodes to exchange data (election, synchronization) with this node

- initial-cluster: used to bootstrap the initial cluster configuration, information about all nodes in the cluster. The format is: name specified by name=address specified by initial-advertise-peer-urls.

- initial-cluster-token: Unique cluster identifier, nodes with the same identifier will be considered as being in a cluster

Here the structure of the cluster is one master and two standby, distributed on three hosts, the IP addresses of the three hosts are 10.20.16.227, 10.20.16.228, 10.20.16.214.
# Note that the names of the three machines cannot be the same
Host 10.20.16.228 etcd is configured as follows

```
cat > /etc/etcd/conf.yaml <<EOF
name: etcd-2
data-dir: /var/lib/etcd/data
listen-client-urls: http://10.20.16.228:12379,http://127.0.0.1:12379
advertise-client-urls: http://10.20.16.228:12379
listen-peer-urls: http://10.20.16.228:12380
initial-advertise-peer-urls: http://10.20.16.228:12380
initial-cluster: etcd-1=http://10.20.16.227:12380,etcd-2=http://10.20.16.228:12380,etcd-3=http://10.20.16.214:12380
initial-cluster-token: etcd-cluster-token
initial-cluster-state: new
EOF
```
Host 10.20.16.214 etcd is configured as follows

```
cat > /etc/etcd/conf.yaml <<EOF
name: etcd-3
data-dir: /var/lib/etcd/data
listen-client-urls: http://10.20.16.214:12379,http://127.0.0.1:12379
advertise-client-urls: http://10.20.16.214:12379
listen-peer-urls: http://10.20.16.214:12380
initial-advertise-peer-urls: http://10.20.16.214:12380
initial-cluster: etcd-1=http://10.20.16.227:12380,etcd-2=http://10.20.16.228:12380,etcd-3=http://10.20.16.214:12380
initial-cluster-token: etcd-cluster-token
initial-cluster-state: new
EOF
```
**Configure etcd's service**
Executed by root user

```
cat > /etc/systemd/system/etcd.service <<EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
User=root
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/bin/etcd --config-file=/etc/etcd/conf.yaml
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF
```
**Start etcd**
Start etcd as a service
Executed by root user

```
systemctl daemon-reload
```
Open the port, the non-default port must remember to open, otherwise systemctl start etcd will fail

```
systemctl enable etcd   #After enable, etcd will start automatically when the host is booted.
```
Execute as antdb user

```
systemctl start etcd    If it's already on at the beginning, then systemctl restart etcd
systemctl status etcd
```
Check the status of etcd cluster

```
systemctl status etcd
```
Expected output.

```
[root@host227 ~]# systemctl status etcd
● etcd.service - Etcd Server
   Loaded: loaded (/etc/systemd/system/etcd.service; disabled; vendor preset: disabled)
   Active: active (running) since Mon 2022-08-01 17:11:38 CST; 4min 3s ago
 Main PID: 25631 (etcd)
    Tasks: 46
   Memory: 31.1M
   CGroup: /system.slice/etcd.service
           └─25631 /bin/etcd --config-file=/etc/etcd/conf.yaml
```
After etcd is configured on all 3 hosts, you can check the status of the etcd cluster

```
etcdctl --endpoints=http://127.0.0.1:12379 member list
```
Expected output

```
[root@host227 ~]# etcdctl --endpoints=http://127.0.0.1:12379 member list
88b36252d4fcd138: name=etcd-2 peerURLs=http://10.20.16.228:12380 clientURLs=http://10.20.16.228:12379 isLeader=false
9ad1f6e716a2ebe6: name=etcd-1 peerURLs=http://10.20.16.227:12380 clientURLs=http://10.20.16.227:12379 isLeader=true
fc652d5e31669223: name=etcd-3 peerURLs=http://10.20.16.214:12380 clientURLs=http://10.20.16.214:12379 isLeader=false
```
**Manage the cluster with Patroni**
The following operations need to be performed on all 3 hosts, modify the configuration according to the actual situation.
**Configure patroni**
Create the file /etc/patroni.yml and configure it as follows.
Executed by root user.

```
cat > /etc/patroni.yml <<EOF
scope: antdb-cluster        #        Naming of clusters
namespace: /antdb/        # Here the default is service, if it is not set, it will be /service/.
name: adbnode_01           #   Node names, which need to be different for each node

restapi:  
  listen: 10.20.16.227:8008              # Change to actual ip
  connect_address: 10.20.16.227:8008    # Change to actual ip

etcd:
  hosts: 10.20.16.227:12379,10.20.16.228:12379,10.20.16.214:12379

bootstrap:
  # this section will be written into Etcd:/<namespace>/<scope>/config after initializing new cluster
  # and all other cluster members will use it as a `global configuration`
  dcs:
    ttl: 15
    loop_wait: 5
    retry_timeout: 5
    maximum_lag_on_failover: 1048576
    master_start_timeout: 300
    synchronous_mode: true
    postgresql:
      use_pg_rewind: true
      parameters:
         listen_addresses: "*"
         port: 40571					# Change to the actual AntDB port number
         wal_level: logical        #patroni's stream replication level requires logical
         log_directory: "pg_log"
         log_destination: "csvlog"
         hot_standby: "on"
         wal_keep_size: 10240
         max_wal_senders: 10

postgresql:
  listen: 10.20.16.227:40571       # Change to actual ip and AntDB port
  connect_address: 10.20.16.227:40571   # Change to actual ip and AntDB port
  data_dir: /data/shan2/pgdata      # Change to the actual data catalog
  bin_dir: /data/shan2/pgsql_adb/bin   # Change to the actual installation directory
  pgpass: /data/shan2/.pgpass   # Modify it to an actual .pgpass file. pgpass cannot be in the same directory as pgdata.
  authentication:								 # Change to actual username and password
    replication:
      username: replicator          
      password: antdb
    superuser:
      username: shan
      password: shan@123
  parameters:
    unix_socket_directories: '/tmp'

tags:
    nofailover: false
    noloadbalance: false
    clonefrom: false
    nosync: false
EOF
```
The above configuration is modified according to the actual situation.
**Configure service for patroni**
Executed by the root user.
Note that the User and Group users here are the same as the AntDB installation users.

```
cat > /usr/lib/systemd/system/patroni.service <<EOF

[Unit]
Description=Runners to orchestrate a high-availability AntDB5.0
After=syslog.target network.target

[Service]
Type=simple
User=shan2
Group=shan2
ExecStart=/usr/local/bin/patroni /etc/patroni.yml
KillMode=process
TimeoutSec=30
Restart=no

[Install]
WantedBy=multi-user.target
EOF
```
Start patroni
Executed by the root user to start patroni as a service
After configuration, you must reload, if you change the configuration in the middle, you must also reload, otherwise it will not be recognized.

```
systemctl daemon-reload 
```
Execute as root user

```
systemctl start patroni   
```
Check the startup status

```
systemctl status patroni –l
```
The following display means the startup is successful.

```
[shan2@host227 ~]$ systemctl status patroni –l
● patroni.service - Runners to orchestrate a high-availability AntDB5.0
   Loaded: loaded (/usr/lib/systemd/system/patroni.service; disabled; vendor preset: disabled)
   Active: active (running) since Fri 2022-08-05 17:40:21 CST; 2 days ago
 Main PID: 179523 (patroni)
    Tasks: 12
   Memory: 54.9M
   CGroup: /system.slice/patroni.service
           ├─179523 /usr/local/bin/python3.7 /usr/local/bin/patroni /etc/patroni.yml
           ├─179905 /data/shan2/pgsql_adb/bin/postgres -D /data/shan2/pgdata --config-file=/data/shan2/pgdata/p...
           ├─179908 postgres: antdb-cluster: startup   recovering 000000040000000000000005
           ├─179910 postgres: antdb-cluster: checkpointer   
           ├─179911 postgres: antdb-cluster: background writer   
           ├─179912 postgres: antdb-cluster: stats collector   
           ├─179917 postgres: antdb-cluster: shan2 postgres 10.20.16.227(56482) idle
           └─179927 postgres: antdb-cluster: walreceiver   streaming 0/50010D0
Unit \xe2\x80\x93l.service could not be found.
```
If the startup fails, you can manually execute the ExecStart configuration in the service configuration file above to manually start the service to see the error report.

```
/usr/local/bin/patroni /etc/patroni.yml
```
Or
The patroni log is in the system log, if there is a problem, you can go through the log error reporting to investigate

```
tail -f /var/log/messages
```
After successful installation, check the status of the cluster.

```
[root@host228 shan2]# patronictl -c /etc/patroni.yml list 
+------------+--------------------+--------------+---------+----+-----------+-----------------+
| Member     | Host               | Role         | State   | TL | Lag in MB | Pending restart |

Cluster: antdb-cluster (7127106860783255508) --+---------+----+-----------+-----------------+
| adbnode_01 | 10.20.16.227:40571 | Sync Standby | running |  4 |         0 | *               |
| adbnode_02 | 10.20.16.228:40571 | Replica      | running |  4 |         0 | *               |
| adbnode_03 | 10.20.16.214:40571 | Leader       | running |  4 |           | *               |
+------------+--------------------+--------------+---------+----+-----------+-----------------+
```
**Some common operations of patroni**
patronictl to view cluster information

```
patronictl -c /etc/patroni.yml list 
```
Restart the cluster, postgres-cluster is the cluster name

```
patronictl -c /etc/patroni.yml restart postgres-cluster
```
Restart the nodes

```
patronictl -c /etc/patroni.yml restart postgres-cluster pgnode02
```
Reinitialize the nodes

```
patronictl -c /etc/patroni.yml reinit postgres-cluster pgnode02
```
Send a SQL command

```
patronictl -c /etc/patroni.yml query postgres-cluster --command 'select version() ' -d postgres

version

PostgreSQL 13.3 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44), 64-bit
```
Get the master node dsn information

```
patronictl -c /etc/patroni.yml dsn postgres-cluster
host=10.19.32.118 port=55551
```
Manually switchover a node

```
patronictl -c /etc/patroni.yml switchover
```
Manually failover a node

```
patronictl -c /etc/patroni.yml failover
```
Delete cluster information in DCS

```
patronictl -c /etc/patroni.yml remove postgres-cluster
```
•Add patroni and database nodes
•Install antdb, patroni on the new node
•Provide authentication of the new node on the corresponding existing node (pg_hba.conf)
•Copy the yml file to the new node
•Modify the yml file of the new node
 

```
name: antdb120
tags: 
replicatefrom: antdb119 ###Select the cascade slave node, the default is to synchronize to the master node 
restapi: 
listen: 10.21.20.120:8008
connect_address: 10.21.20.120:8008 
postgresql: 
listen: 0.0.0.0:6432 
connect_address: 10.21.20.120:6432
data_dir: /home/antdb/datafile
```
•Start patroni
 

```
sudo systemctl status patroni -l
sudo journalctl -f -u patroni 
```
 
 The patroni log is in the system log, if there is a problem, you can go through the log error reporting to investigate
 

```
tail -f /var/log/messages
```
**Build high availability under ipv6**
The steps to build high availability under Ipv6 are the same as the overall steps above, except that the ip address part needs to be changed, as described below.