AntDB Installation and Deployment Documentation - P2
Tags: database, datascience
Published: 2023-07-12 09:00:52

## Software Installation and Deployment

This section describes how to install the AntDB Distributed Edition.

### Distributed Edition (Single Center) Installation and Deployment<a name='addlocal'></a>

#### Installation using RPM packages (not recommended)

##### Dependency package installation

For offline environment, it is recommended to build a local yum source on the host computer through the image file of the operating system to facilitate the installation of dependencies. Under cento or redhat operating system, execute the following command to install the dependencies.

```
sudo yum install -y perl-ExtUtils-Embed
sudo yum install -y flex
sudo yum install -y bison
sudo yum install -y readline-devel
sudo yum install -y zlib-devel
sudo yum install -y openssl-devel
sudo yum install -y pam-devel
sudo yum install -y libxml2-devel
sudo yum install -y libxslt-devel
sudo yum install -y openldap-devel
sudo yum install -y python-devel
sudo yum install -y gcc-c++ 
```

For other operating systems, please just change the installation command, and the dependency package names will remain the same. Some dependencies do not provide libssh2 package, you can install it by compiling the source code.

* Download at: https://www.libssh2.org/download/
* Source code compilation and installation (executed by root).

```
wget https://www.libssh2.org/download/libssh2-1.9.0.tar.gz
tar xzvf libssh2-1.9.0.tar.gz
cd libssh2-1.9.0
./configure
make
make install
```

##### Install RPM packages

Upload the RPM package provided by the delivery person to the server on the host where ADBMGR is located and install it under root or a user with sudo privileges. Install to the default path /opt/app/antdb.

```
sudo rpm -ivh antdb-xxx.rpm
```

If you want to install to other paths, you can do so as follows:

```
sudo rpm -ivh antdb-xxx.rpm --relocate=/opt/app/antdb=$ADBHOME
```

$ADBHOME as a custom directory, the name is best to be known by its name, e.g.:

```
/home/antdb/app/antdb
```

After the RPM package is installed, the value of the variable ADB\_HOME will either be /opt/app/antdb, or a directory of your choice. Next, you need to modify the directory permissions:

```
sudo chown -R antdb:antdb $ADBHOME
sudo chmod -R 755 $ADBHOME
```

##### Configure Environment Variables

After installing the RPM package, the initmgr and mgr\_ctl executables will be generated in the bin directory of the specified directory (i.e. $ADBHOME directory). To initialize adbmgr you also need to configure the PATH variable to do so, by appending the following to the hidden file .bashrc under the current user (execute vim ~/.bashrc to open the file):

```
export ADBHOME=/opt/app/antdb
export PATH=$ADBHOME/bin:$PATH
export LD_LIBRARY_PATH=$ADBHOME/lib:$LD_LIBRARY_PATH
export PGDATABASE=postgres

export mgrdata=/data/antdb/mgr1
alias adbmgr='psql -p 6432 -d postgres '
alias mgr_stop='mgr_ctl stop -D $mgrdata -m fast'
alias mgr_start='mgr_ctl start -D $mgrdata' 
```

Note: ADBHOME needs to be set according to the path of the binary executable generated by the compilation of AntDB. mgrdata also needs to be modified according to the actual path of the respective environment. Then execute the following command to make it effective:

```
source ~/.bashrc
```

#### Install with tarball

Copy the installation package (ADB-Community.tar.gz) to the directory of AntDB user (e.g. uatadb) and change the permissions of the package.

```
chown -R uatadb:uatadb ADB-Community.tar.gz #Change the user permission of the installer to uatadb
```

Login to the new user to install AntDB

```
su - uatadb                           #Login to the newly created AntDB user
tar -xzvf ADB-Community.tar.gz    #Unzip the installation package
cd ADB-Community                  #Enter the extracted directory ADB-Community with the cd command 
```

##### Interactive installation

Execute sh AntDB\_install.sh under ADB-Community to install AntDB interactively.

```
sh AntDB_install.sh
```

The interactive installation can be customized in three ways, as shown below. You can choose the default value or customize it. However, it is important to note that you must select "d" for the second place, which means that the installation is a distributed database.

```
# Step one:
# Select whether you agree to the user agreement, only if you agree (Y) can you continue to install further, if you do not agree (N), the installation will stop.
Do you agree?(Y/N)[Default:Y]
You input:
OK, you agree to the agreement. continue...
check_agreement...success.

# Step two:
# Select the mode which you want to install, centralized (c), distributed (d). The default selection is c, for a centralized installation. Enter Q to abort this installation.
# Here, we choose d for a distributed database installation
Please enter the deployment mode "(c)entralized" or "(d)istributed" or "Q" to exit the installation.[Default:centralized]d
You input : d

# Step three:
# Fill in the installation path. Note: You need to fill in the absolute path.
# If the user does not enter the absolute path, just enter, it will be installed to the default path. Entering Q aborts this installation.
Please enter a valid empty directory to install antdb. Or enter "Q/q" to exit the installation.[Default:/data/adb01/app/antdb]
You input :
Use default install path:[/data/adb01/app/antdb]. continue...
check install path...
check install path ...success.
decompress_package...
decompress_package...success.
copy files...
copy files...success.
set bash environment...
please excute cmd 'source /data/adb01/.bashrc' to make the environment variables take effect.
set bash environment...success.
set kernel config...
Please refer to the following files in the installation directory to manually set the recommended kernel config.
/data/adb01/app/antdb/setup/common/kernel_limits.conf
/data/adb01/app/antdb/setup/common/kernel_sysctl.conf
antdb install ...success.
```

After the above execution, according to the prompt, you need to source the environment variables, and after the source is done, the installation of the binaries is finished.

```
source /data/adb01/.bashrc
```

#### Deploy adbmgr

1.Initialize adbmgr.

```
initmgr -D $mgrdata
```

Where $mgrdata is the user-specified installation directory for adbmgr (this file directory needs to be created in advance). After initialization, the following files are generated in the specified directory.

```
[gd@INTEL175 ~]$ cd /data/antdb/mgr1 
[gd@INTEL175 mgr1]$ ll
total 112
drwx------ 5 gd gd  4096 Oct 17 17:28 base
drwx------ 2 gd gd  4096 Oct 17 17:28 global
drwx------ 2 gd gd  4096 Oct 17 17:28 pg_clog
drwx------ 2 gd gd  4096 Oct 17 17:28 pg_commit_ts
drwx------ 2 gd gd  4096 Oct 17 17:28 pg_dynshmem
-rw------- 1 gd gd  4450 Oct 17 17:28 pg_hba.conf
-rw------- 1 gd gd  1636 Oct 17 17:28 pg_ident.conf
drwx------ 4 gd gd  4096 Oct 17 17:28 pg_logical
drwx------ 4 gd gd  4096 Oct 17 17:28 pg_multixact
drwx------ 2 gd gd  4096 Oct 17 17:28 pg_notify
drwx------ 2 gd gd  4096 Oct 17 17:28 pg_replslot
drwx------ 2 gd gd  4096 Oct 17 17:28 pg_serial
drwx------ 2 gd gd  4096 Oct 17 17:28 pg_snapshots
drwx------ 2 gd gd  4096 Oct 17 17:28 pg_stat
drwx------ 2 gd gd  4096 Oct 17 17:28 pg_stat_tmp
drwx------ 2 gd gd  4096 Oct 17 17:28 pg_subtrans
drwx------ 2 gd gd  4096 Oct 17 17:28 pg_tblspc
drwx------ 2 gd gd  4096 Oct 17 17:28 pg_twophase
-rw------- 1 gd gd     4 Oct 17 17:28 PG_VERSION
drwx------ 3 gd gd  4096 Oct 17 17:28 pg_xlog
-rw------- 1 gd gd    88 Oct 17 17:28 postgresql.auto.conf
-rw------- 1 gd gd 22234 Oct 17 17:28 postgresql.conf
```

2.Modify the adbmgr configuration parameters.

a. Modify postgresql.conf.

```
cat >> ${mgrdata}/postgresql.conf << EOF
port = 6432
listen_addresses = '*'
log_directory = 'pg_log'
log_destination ='csvlog'
logging_collector = on
log_min_messages = error
max_wal_senders = 3
hot_standby = on
wal_level = replica
EOF
```

The default port for adbmgr is 6432, and the user can set an available port according to the planning. 

b. Modify pg\_hba.conf.

```
cat >> ${mgrdata}/pg_hba.conf << EOF
host replication all 10.0.0.0/8 trust
host all all 10.0.0.0/8 trust
EOF
```

The IP in hba needs to be modified according to the actual situation. 

3.Start adbmgr; After adbmgr is initialized successfully, you can start it.

```
mgr_ctl start -D /data/antdb/mgr1
```

/data/antdb/mgr To be consistent with the initialization path, both are the values of the variable mgrdata.

```
[gd@INTEL175 ~]$ mgr_ctl start -D /data/antdb/mgr
server starting
[gd@INTEL175 ~]$ LOG: database system was shut down at 2017-10-17 17:38:15 CST
LOG: MultiXact member wraparound protections are now enabled
LOG: database system is ready to accept connections
LOG: autovacuum launcher started
LOG: adb monitor launcher started
```

If adbmgr reports that the port is occupied during startup, the reason is that the current port number is already used by another process (adbmgr uses port 6432 by default). You need to change the value of the variable port in /data/antdb/mgr/postgresql.conf file so that it does not conflict with other processes. After starting, ps can view the processes of mgr as follows.

```
67456 ?       S      0:00 /data/antdb/app/bin/adbmgrd -D mgr
67458 ?       Ss     0:00 \_ adbmgr: checkpointer process
67459 ?       Ss     0:00 \_ adbmgr: writer process    
67460 ?       Ss     0:00 \_ adbmgr: wal writer process    
67461 ?       Ss     0:00 \_ adbmgr: autovacuum launcher process   
67462 ?       Ss     0:00 \_ adbmgr: stats collector process   
67463 ?       Ss     0:00 \_ adbmgr: adb monitor launcher process
```

4.Configure standby adbmgr (optional); In a real production environment, adbmgr also needs to be highly available, so it is also necessary to configure a standby for adbmgr. The standby for adbmgr may not be added during the feature verification phase, and will be added later when high availability is verified. On the selected host where the adbmgr standby is deployed, use the antdb user to execute:

```
pg_basebackup -h 10.20.16.175 -p 6432 -U antdb -D /data/antdb/mgr --
nodename mgr -Xs -Fp -R
```

Where:

* -h adb05 in adb05 is the hostname of the adbmgr master, where the IP can also be used.
* -p 6432 for the port of the adbmgr master.
* -U antdb means use antdb user to connect to adbmgr master.
* -D specifies the data directory of the adbmgr slave on the execution host, ensuring that the permissions are correct.
* Other parameters remain the same; after execution is complete, start adbmgr slave.

```
 mgr_ctl start -D /data/antdb/mgr_standby
```

The value of adbhome here should match the value of ADBHOME in .bashrc.

#### Add hosts and deploy binaries<a name="addhost"></a>

1.Add host; add several hosts according to the planning, the cluster is deployed on several hosts.

```
add host hostname (port=22,protocol='ssh',adbhome='AntDB installation path coresponding to the host]',address='server IP',agentport=agent port number,user=host user);
```

* port: ssh port number, default 22.
* protocol: login protocol, default ssh, users can leave it out.
* adbhome: installation directory of antdb, the next deploy binary will copy the binary on mgr to this directory.
* address: host IP address (IPV4, IPV6 and domain names are supported).
* agentport: since an agent process will be started on each host, here is the corresponding port number.
* user: ssh login username; For example:

```
--IPv4
add host host227(port=22,protocol='ssh',adbhome='/home/uatadb/app',address='10.23.16.227',agentport=31001,user='uatadb');
add host host228(port=22,protocol='ssh',adbhome='/home/uatadb/app',address='10.23.16.228',agentport=31001,user='uatadb');

--IPv6
add host host227(port=22,protocol='ssh',adbhome='/home/uatadb/app',address='fc00::227',agentport=31001,user='uatadb');
add host host228(port=22,protocol='ssh',adbhome='/home/uatadb/app',address='fc00::228',agentport=31001,user='uatadb');

--doamin name
add host host227(port=22,protocol='ssh',adbhome='/home/uatadb/app',address='adb227.host.com',agentport=31001,user='uatadb');
add host host228(port=22,protocol='ssh',adbhome='/home/uatadb/app',address='adb228.host.com',agentport=31001,user='uatadb');
```

2.deploy binary program; The deploy command will package the AntDB binary executable files and send it to all hosts in the host table. If mutual trust is not set among the hosts in the cluster, deploy all requires user password (current user's login password), if mutual trust is set among the hosts, the tedious setting of password can be eliminated.

* Deploy all hosts at once:

```
deploy all password '123456';
```

* Deploy the specified hosts, for example, deploy the specified hosts adb01 and adb02:

```
deploy host227,host228 password '123456';
```

3.Start agent. There are two ways: start all the agents at once and start the specified host agent. Note: password is the linux system password corresponding to the host user in the host table, which is used to communicate with the host, not the user password of the AntDB database. When the password starts with a number, you need to add single quotes or double quotes, for example, password '12345z' is correct, password 12345z will report an error; if the password does not start with a number, you can add quotes or not.

* Start all agents at once:

```
start agent all password '123456';
```

* Start the specified agent, multiple hosts need to be executed multiple times, e.g.:

```
start agent host227,host228 password '123456';
start agent host227 password '123456';
```

When the password is incorrect and the start of the agent fails, the following error is reported:

```
postgres=# start agent host227 password '123456abc'; 
hostname | status | description                
------------+--------+------------------------------------------- 
adb01 | f | Authentication failed (username/password)
(1 row) 
```

#### Add Nodes

Add node information such as gtmcoord, coordinator, datanode master, datanode slave in Node table. Note: host name must be from host table, port number should not conflict, and the folder specified by path must be empty, otherwise the initialization will fail and error will be reported. This setting is to prevent users from operating and forgetting that there is useful data information under the current node. Add command:

1.add gtmcoord information. The slave node must have a different name from the master node, so the specified master must exist, and the same asynchronous relationship is set by the SYNC\_STATE parameter.

```
add gtmcoord master name(host='localhost3',port=xxx, path='xxx');add gtmcoord slave name for maste_name(host='localhost2',port=xxx, path='xxx'
```

2.add coordinator node.

```
add coordinator master name(path = 'xxx', host='localhost1', port=xxx);
```

3.add datanode master node.

```
add datanode master name(path = 'xxx', host='localhost1', port=xxx);
```

4.add datanode slave node add datanode slave information, the slave node must have a different name from the master, so the specified master must exist, and the same asynchronous relationship is set by the SYNC\_STATE parameter.

```
add datanode slave name for master_name (host='localhost2', port=xxx, path='xxx', SYNC_STATE='sync');
```

After adding, use the command list node to view the information of the node just added. A complete example: (gtmcoord/datanode are one master and one slave)

```
add host adb01(port=22,protocol='ssh',adbhome='/opt/app/antdb',address="10.1.226.201",agentport=8432,user='antdb');
add host adb02(port=22,protocol='ssh',adbhome='/opt/app/antdb',address="10.1.226.202",agentport=8432,user='antdb'); 
add host adb03(port=22,protocol='ssh',adbhome='/opt/app/antdb',address="10.1.226.203",agentport=8432,user='antdb'); 
deploy all password '123456';
start agent all password '123456';
add gtmcoord master gc_1(host='adb03',port=6655, path='/home/antdb/data/gc_1'); 
add gtmcoord slave gc_2 for gc_1(host='adb02',port=6655,path='/home/antdb/data/gc_2'); 
add coordinator master cn1(host='adb01', port=5432,path = '/home/antdb/data/cn1'); 
add coordinator master cn2(host='adb02', port=5432,path = '/home/antdb/data/cn2'); 
add datanode master dn1_1(host='adb01', port=14332,path = '/home/antdb/data/dn1_1'); 
add datanode slave dn1_2 for dn1_1 (host='db02',port=14332,path='/home/antdb/data/dn1_2'); 
add datanode master dn2_1(host='adb02', port=24332,path = '/home/antdb/data/dn2_1'); 
add datanode slave dn2_2 for dn2_1(host='adb01',port=24332,path='/home/antdb/data/dn2_2'); 
```

Recommendations on node deployment for production environments:

* At most one coordinator on each host, with the same port for each coordinator.
* Because the application can also connect to gtmcoord, the host with gtmcoord can no longer deploy the coordinator during deployment.
* the ports of the datanode can use the same port on different hosts according to a group of nodes (one master and multiple slaves as a group, e.g. dn1\_x).

#### Initialize the cluster

The cluster nodes have all been added, so you can use the command init all to initialize the cluster. As shown below, you can see the steps inside init all.

```
postgres=# init all ;
  operation type     | nodename | status | description 
-----------------------+----------+--------+------------- 
init gtmcoord master | gc_1     | t     | success 
start gtmcoord master | gc_1     | t     | success 
init coordinator     | cn1       | t     | success 
init coordinator     | cn2       | t     | success 
start coordinator     | cn1       | t     | success 
start coordinator     | cn2       | t     | success 
init datanode master | dn1_1     | t     | success 
init datanode master | dn2_1     | t     | success 
start datanode master | dn1_1     | t     | success 
start datanode master | dn2_1     | t     | success 
init gtmcoord slave   | gc_2     | t     | success 
start gtmcoord slave | gc_2     | t     | success 
init datanode slave   | dn1_2     | t     | success 
init datanode slave   | dn1_2     | t     | success 
start datanode slave | dn2_2     | t     | success 
start datanode slave | dn2_2     | t     | success 
config coordinator   | cn1       | t     | success 
config coordinator   | cn2       | t     | success 
(18 rows) 
```

If there is an error in the middle of init all and some nodes fail to initiate and you want to reinitialize, you can execute the following command to clean up the environment and reinitialize.

```
stop all mode i; --Stop all nodes
clean all; --Empty the node directory
init all; --re-initialize
```

View the operation status of each node of the cluster through monitor all:

```
postgres=# monitor all ; 
nodename |   nodetype           | status | description |     host     | port | recovery | boot time                       |     nodezone  
----------+-----------------+--------+-------------+--------------+------------ --------+----- --------+-------------+--------------+------------ 
cn1     | coordinator master   | t     | running     | 10.1.226.201 |  5432 | false     | 2020-09-01 10:09:30.123+08     | local  
cn2     | coordinator master   | t     | running     | 10.1.226.202 |  5432 | false     | 2020-09-01 10:09:30.123+08     | local      
dn1_1   | datanode master       | t     | running     | 10.1.226.201 | 14332 | false     | 2020-09-01 10:09:30.123+08     | local      
dn1_2   | datanode slave       | t     | running     | 10.1.226.202 | 14332 | true       | 2020-09-01 10:09:30.123+08     | local      
dn2_1   | datanode master       | t     | running     | 10.1.226.202 | 24332 | false     | 2020-09-01 10:09:30.123+08     | local      
dn2_2   | datanode slave       | t     | running     | 10.1.226.201 | 24332 | true       | 2020-09-01 10:09:30.123+08     | local      
gc_1     | gtmcoord master       | t     | running     | 10.1.226.203 |  6655 | false     | 2020-09-01 10:09:30.123+08     | local      
gc_2     | gtmcoord slave       | t     | running     | 10.1.226.202 |  6655 | true       | 2020-09-01 10:09:30.123+08     | local      
(8 rows) 
```

At this point, AntDB cluster initialization is completed. The recovery field indicates whether the node is in recovery state, normally, the value of master node is false, and the value of slave node is true. Note: init all will include the start all operation, which is only used when initializing the cluster, if you want to start the cluster later, please use the start all command. You can log in to the gtmcoord master or the coord master node via the psql command:

```
psql -p 5432 #log into coord
psql -p 6655 #log into gtmcoord 
```

Note: The application needs to connect to gtmcoord or coord for operations such as adding, deleting, and checking, not adbmgr, which is only used to manage the cluster.

#### Generated files list description

The list of files generated after AntDB installation is completed is as follows:

| File name| Permission| Function
|----------|----------|----------
| base| drwx------| Subdirectories of each database for the subdirectory
| global| drwx------| A subdirectory containing cluster-wide tables, such as pg\_database
| logfile| -rw-------| Startup Log
| pg\_commit\_ts| drwx------| Subdirectory containing transaction commit timestamp data
| pg\_dynshmem| drwx------| Subdirectory containing files used by the dynamic shared memory subsystem
| pg\_hba.conf| -rw-------| Client authentication control filecontrol file
| pg\_ident.conf| -rw-------| Used to configure which operating system users can be mapped as database users
| pg\_log| drwx------| pg run log
| pg\_logical| drwx------| Subdirectory containing state data for logical replication
| pg\_multixact| drwx------| Subdirectory containing multi-transaction state data (user shared row locks)
| pg\_notify| drwx------| Subdirectories containing replication slot data
| pg\_replslot| drwx------| Subdirectory containing LISTEN/NOTIFY status data
| pg\_rxlog| drwx------| Subdirectory containing transaction commit status data
| pg\_serial| drwx------| A subdirectory containing information about committed serializable transactions
| pg\_snapshots| drwx------| Subdirectories containing snapshots of everywhere
| pg\_stat| drwx------| Subdirectory containing permanent files for the statistics subsystem
| pg\_stat\_tmp| drwx------| Subdirectory containing temporary files for the statistical information subsystem
| pg\_subtrans| drwx------| Subdirectories containing subtransaction status data
| pg\_tblspc| drwx------| Subdirectory containing symbolic links to tablespaces