Manager Handbook for Distributed AntDB-T - P19
Published: 2023-08-23 02:16:11

Host/Standby Switchover
In the latest version of AntDB, high availability is done automatically by the self-healing module, so users don’t need to pay attention to it. So the content of this section is only for reference when users perform it manually.
There are 3 commands involved in host/standby switchover, namely failover, rewind and switchover, which are applied to different scenarios.
Update standby to host and remove host
When the host node is down, the command failover is used to switch the nodes of the host and standby of datanode, coordinator and gtmcoord nodes.

```
postgres=# failover datanode dn1_1;  
WARNING:  An exception occurred during the switching operation, It is recommended to use command such as 'monitor all', 'monitor ha' to check the failure point in the cluster first, and then retry the switching operation!!!  
ERROR:  Can't find a Synchronous standby node, Abort switching to avoid data loss  
postgres=# failover datanode dn1_1 force;  
NOTICE:  dn1_3 have the best wal lsn, choose it as a candidate for promotion  
NOTICE:  gcn1 try lock cluster successfully  
NOTICE:  dn1_3 was successfully promoted to the new master  
NOTICE:  dn1_3 running on correct status of master mode  
NOTICE:  set GTM information on dn1_2 successfully  
NOTICE:  dn1_2 running on correct status of slave mode  
NOTICE:  dn1_1 is waiting for rewinding. If the doctor is enabled, the doctor will automatically rewind it  
NOTICE:  gcn1 try unlock cluster successfully  
NOTICE:  Switch the datanode master from dn1_1 to dn1_3 has been successfully completed  
 nodename | status |  description   
----------+--------+-------------------  
 dn1_3  | t  | promotion success  
(1 row)
```
As we can see from the operation, if the host node has no synchronous slave, the failover operation is not allowed, otherwise it will cause data loss, and you need to use the optionforce to force the switchover.

Host/Standby Switchover

When the user needs to switch the original host to the standby and the original standby to the host, the switchover command can be used.

```
postgres=# switchover datanode slave dn2_2;  
NOTICE:  wait max 10 seconds to wait there are no active connections on coordinators and master nodes  
NOTICE:  gcn1 try lock cluster successfully  
NOTICE:  wait max 10 seconds to check there are no active locks in pg_locks except the locks on pg_locks table  
NOTICE:  wait max 10 seconds to check dn2_1 and dn2_2 have the same xlog position  
NOTICE:  dn2_2 was successfully promoted to the new master  
NOTICE:  dn2_2 running on correct status of master mode  
NOTICE:  dn2_1 running on correct status of slave mode  
NOTICE:  gcn1 try unlock cluster successfully  
NOTICE:  Switch the datanode master from dn2_1 to dn2_2 has been successfully completed  
 nodename | status |  description   
----------+--------+-------------------  
 dn2_2  | t  | promotion success  
(1 row)  
 
postgres=# list node dn2_2;  
 name  | host  |  type  | mastername | port  | sync_state |  path  | initialized | incluster | readonly | zone   
-------+-------+-----------------+------------+-------+------------+----------------------------------+-------------+-----------+----------+-------  
 dn2_2 | adb01 | datanode master |  | 52542 |  | /data/antdb/data/adb50/d2/dn2_2 | t  | t  | f  | local  
(1 row)  
postgres=# list node dn2_1;  
 name  | host  |  type  | mastername | port  | sync_state |  path  | initialized | incluster | readonly | zone   
-------+-------+----------------+------------+-------+------------+----------------------------------+-------------+-----------+----------+-------  
 dn2_1 | adb01 | datanode slave | dn2_2  | 52541 | sync  | /data/antdb/data/adb50/d1/dn2_1 | t  | t  | f  | local  
(1 row)  
```
As you can see, the switchover occurs between the host and the standby.

Quick recovery of the standby node

If the master fails, the failover command is executed, and the slave is upgraded to master, the user can have two options when he wants to re-add a new slave to the new master.

Method 1: add a brand new slave node using the append command, which requires data synchronization to ensure that the master and backup are consistent.

Method 2: Add the previously removed master to the cluster as a slave node.

The second way takes a short time because the original master node already has a lot of data, only a small amount of data synchronization is needed after rejoining the cluster. This is exactly what rewind can do to quickly restore the backup node. You can refer to the rewind chapter for details.

Note: To use the rewind feature, you need to set wal_log_hints and full_page_writes to on on the datanode.

```
set datanode all(wal_log_hints=on, full_page_writes=on);  
rewind datanode slave dm1;  
```
For example:

```
postgres=# add datanode slave dn1_1 for dn1_3 (host=adb01,port=52531,path='/data/antdb/data/adb50/d1/dn1_1');  
ADD NODE  
postgres=# rewind datanode slave dn1_1;  
NOTICE:  pg_ctl restart datanode slave "dn1_1"  
NOTICE:  10.21.20.175, pg_ctl  restart -D /data/antdb/data/adb50/d1/dn1_1 -Z datanode -m fast -o -i -w -c -l /data/antdb/data/adb50/d1/dn1_1/logfile  
NOTICE:  wait max 90 seconds to check datanode slave "dn1_1" running normal  
NOTICE:  pg_ctl stop datanode slave "dn1_1" with fast mode  
NOTICE:  10.21.20.175, pg_ctl  stop -D /data/antdb/data/adb50/d1/dn1_1 -Z datanode -m fast -o -i -w -c  
NOTICE:  wait max 90 seconds to check datanode slave "dn1_1" stop complete  
NOTICE:  update gtmcoord master "gcn1" pg_hba.conf for the rewind node dn1_1  
NOTICE:  update gtmcoord slave "gcn2" pg_hba.conf for the rewind node dn1_1  
NOTICE:  update datanode master "dn1_3" pg_hba.conf for the rewind node dn1_1  
NOTICE:  update datanode slave "dn1_2" pg_hba.conf for the rewind node dn1_1  
NOTICE:  on datanode master "dn1_3" execute "checkpoint"  
NOTICE:  10.21.20.175,  /data/antdb/app/adb50/bin/pg_controldata '/data/antdb/data/adb50/d2/dn1_3' | grep 'Minimum recovery ending location:' |awk '{print $5}'  
NOTICE:  receive msg: {"result":"0/0"}  
NOTICE:  10.21.20.175,  /data/antdb/app/adb50/bin/pg_controldata '/data/antdb/data/adb50/d2/dn1_3' |grep 'Min recovery ending loc' |awk '{print $6}'  
NOTICE:  receive msg: {"result":"0"}  
NOTICE:  10.21.20.175, adb_rewind  --target-pgdata /data/antdb/data/adb50/d1/dn1_1 --source-server='host=10.21.20.175 port=52533 user=antdb dbname=postgres' -T dn1_1 -S dn1_3  
NOTICE:  receive msg: servers diverged at WAL location 0/40001B0 on timeline 1  
rewinding from last common checkpoint at 0/4000140 on timeline 1  
Done!  
NOTICE:  refresh mastername of datanode slave "dn1_1" in the node table  
NOTICE:  set parameters in postgresql.conf of datanode slave "dn1_1"  
NOTICE:  refresh recovery.conf of datanode slave "dn1_1"  
NOTICE:  pg_ctl  start -Z datanode -D /data/antdb/data/adb50/d1/dn1_1 -o -i -w -c -l /data/antdb/data/adb50/d1/dn1_1/logfile  
NOTICE:  10.21.20.175, pg_ctl  start -Z datanode -D /data/antdb/data/adb50/d1/dn1_1 -o -i -w -c -l /data/antdb/data/adb50/d1/dn1_1/logfile  
NOTICE:  refresh datanode master "dn1_3" synchronous_standby_names='1 (dn1_2,dn1_1)'  
 mgr_failover_manual_rewind_func   
---------------------------------  
 t  
(1 row)  
postgres=# list node dn1_3;  
 name  | host  |  type  | mastername | port  | sync_state |  path  | initialized | incluster | readonly | zone   
-------+-------+-----------------+------------+-------+------------+----------------------------------+-------------+-----------+----------+-------  
 dn1_3 | adb01 | datanode master |  | 52533 |  | /data/antdb/data/adb50/d2/dn1_3 | t  | t  | f  | local  
(1 row)  
postgres=# list node dn1_1;  
 name  | host  |  type  | mastername | port  | sync_state |  path  | initialized | incluster | readonly | zone   
-------+-------+----------------+------------+-------+------------+----------------------------------+-------------+-----------+----------+-------  
 dn1_1 | adb01 | datanode slave | dn1_3  | 52531 | potential  | /data/antdb/data/adb50/d1/dn1_1 | t  | t  | f  | local  
(1 row)  
```
Again, in the latest version, if the self-healing doctor is configured, none of the above operations require manual intervention.