Managing High Availability in PostgreSQL – Part III: Patroni
Tags: postgres, tutorial, python
Published: 2019-08-22 18:08:02

---
title: Managing High Availability in PostgreSQL – Part III: Patroni
published: true
description: Best PostgreSQL high availability frameworks server test scenarios: PostgreSQL Automatic Failover PAF vs. Replication Manager repmgr vs. Patroni: ScaleGrid Blog
tags: PostgreSQL, Open Source, Tutorial, Python,
canonical_url: https://scalegrid.io/blog/managing-high-availability-in-postgresql-part-3/
---
<link rel="canonical" href="https://scalegrid.io/blog/managing-high-availability-in-postgresql-part-3/" />

<a href="https://scalegrid.io/blog/managing-high-availability-in-postgresql-part-3/"><img src="https://scalegrid.io/blog/wp-content/uploads/2019/08/Managing-High-Availability-in-PostgreSQL-Part-3-Patroni-ScaleGrid-Blog.png" alt="Managing High Availability in PostgreSQL – Part III: Patroni - ScaleGrid Blog"></a>
<p style="text-align: justify; padding-bottom:15px;">In our previous blog posts, we discussed the capabilities and functioning of PostgreSQL Automatic Failover (PAF) by Cluster Labs and Replication Manager (repmgr) by 2ndQuadrant. In the final post of this series, we will review the last solution, Patroni by Zalando, and compare all three at the end so you can determine which high availability framework is best for your <a href="https://scalegrid.io/postgresql.html" target="_blank" style="color:#2da964;">PostgreSQL hosting</a> deployment.</p>

<ul>
<li style="padding-bottom:10px;"><a href="https://scalegrid.io/blog/managing-high-availability-in-postgresql-part-1/" target="_blank" style="color:#2da964;">Managing High Availability in PostgreSQL – Part I: PostgreSQL Automatic Failover</a></li>
	<li><a href="https://scalegrid.io/blog/managing-high-availability-in-postgresql-part-2/" target="_blank" style="color:#2da964;">Managing High Availability in PostgreSQL – Part II: Replication Manager</a></li>
<li style="padding-bottom:10px;"><a href="https://scalegrid.io/blog/managing-high-availability-in-postgresql-part-3/" target="_blank" style="color:#2da964;">Managing High Availability in PostgreSQL – Part III: Patroni</a></li>	
</ul>
<h2 style="padding-top: 15px;">Patroni for PostgreSQL</h2>
<p style="text-align: justify;"><a href="https://patroni.readthedocs.io/en/latest/" target="_blank" style="color:#2da964;">Patroni</a> originated as a fork of Governor, a project from Compose. It is an open-source tool suite, written in Python, for managing high availability of PostgreSQL clusters. Instead of building its own consistency protocol, Patroni smartly leverages the consistency model provided by a <a href="https://patroni.readthedocs.io/en/latest/dynamic_configuration.html" target="_blank" style="color:#2da964;">Distributed Configuration Store</a> (DCS). It also supports other DCS solutions like Zookeeper, etcd, Consul and Kubernetes.</p>
<p style="text-align: justify;">Patroni ensures the end-to-end setup of PostgreSQL HA clusters, including streaming replication. It supports various ways for creating a standby node, and works like a template that can be customized to your needs.</p>
<p style="text-align: justify;">This feature-rich tool exposes its functionality via REST APIs and also via a command line utility called <a href="https://github.com/zalando/patroni/blob/master/patronictl.py" target="_blank" style="color:#2da964;">patronictl</a>. It supports integration with HAProxy by using its health check APIs to handle load balancing.</p>
<p style="text-align: justify;">Patroni also supports event notification with the help of callbacks, which are scripts triggered by certain actions. It enables users to perform any maintenance actions by providing pause/resume functionality. The Watchdog support feature makes the framework even more robust.</p>

<h2 style="padding-top: 15px;">How it Works</h2>
<p style="text-align: justify;">Initially, PostgreSQL and Patroni binaries needs to be installed. Once this is done, you will also need to setup a HA DCS configuration. All the necessary configurations to bootstrap the cluster needs to be specified in the yaml configuration file and Patroni will use this file for initialization. On the first node, Patroni initializes the database, obtains the leader lock from DCS, and ensures the node is being run as the master.</p>
<p style="text-align: justify;">The next step is adding standby nodes, for which Patroni provides multiple options. By default, Patroni uses <a href="https://www.postgresql.org/docs/10/app-pgbasebackup.html" target="_blank" style="color:#2da964;">pg_basebackup</a> to create the standby node, and also supports custom methods like WAL-E, pgBackRest, Barman and others for the standby node creation. Patroni makes it very simple to add a standby node, and handles all the bootstrapping tasks and setting up of your streaming replication.</p>

<p style="text-align: justify;">Once your cluster setup is complete, Patroni will actively monitor the cluster and ensure it's in a healthy state. The master node renews the leader lock every ttl second(s) (default: 30 seconds). When the master node fails to renew the leader lock, Patroni triggers an election, and the node which will obtain the leader lock will be elected as the new master.</p>

<h3 style="padding-top: 15px;">How Does it Handle the Split Brain Scenario?</h3>
<p style="text-align: justify;">In a distributed system, consensus plays an important role in determining consistency, and Patroni uses DCS to attain consensus. Only the node that holds the leader lock can be the master and the leader lock is obtained via DCS. If the master node doesn’t hold the leader lock, then it will be demoted immediately by Patroni to run as a standby. This way, at any point in time, there can only be one master running in the system.</p>

<h3 style="padding-top: 15px;">Are There Any Setup Requirements?</h3>
<ul>
	<li>Patroni needs <a href="https://www.python.org/downloads/" target="_blank" style="color:#2da964;">python 2.7</a> and above.</li>
	<li>The DCS and its specific python module must be installed. For test purposes, DCS can be installed on same nodes running PostgreSQL. However, in production, DCS must be installed on separate nodes.</li>
	<li>The yaml configuration file must be present using these high level configuration settings:
<table class="table table-bordered" style="height: auto; margin-top: 20px;" width="100%">
<tbody>
<tr>
<td style="padding:20px 20px 10px; width: 100%; vertical-align: middle; background-color: #d9fce9;">
<p style="text-align: justify; padding-bottom:6px;"><strong>Global/Universal</strong><br>
This includes configuration such as the name of the host (name) which needs to be unique for the cluster, the name of the cluster (scope) and path for storing config in DCS (namespace).</p>
<p style="text-align: justify; padding-bottom:6px;"><strong>Log</strong><br>
Patroni-specific log settings including level, format, file_num, file_size etc.</p>
<p style="text-align: justify; padding-bottom:6px;"><strong>Bootstrap configuration</strong><br>
This is the global configuration for a cluster that will be written to DCS. These configuration parameters can be changed with the help of Patroni APIs or directly in DCS.The bootstrap configuration includes standby creation methods, initdb parameters, post initialization script etc. It also contains timeouts configuration, parameters to decide the usage of PostgreSQL features like replication slots, synchronous mode etc.This section will be written into /&lt;namespace&gt;/&lt;scope&gt;/config of a given configuration store after the initializing of new cluster.</p>
<p style="text-align: justify; padding-bottom:6px;"><strong>PostgreSQL</strong><br>
This section contains the PostgreSQL-specific parameters like authentication, directory paths for data, binary and config, listen ip address etc.</p>
<p style="text-align: justify; padding-bottom:6px;"><strong>REST API</strong><br>
This section includes the Patroni-specific configuration related to REST API's such as listen address, authentication, SSL etc.</p>
<p style="text-align: justify; padding-bottom:6px;"><strong>Consul</strong><br>
Settings specific to Consul DCS.</p>
<p style="text-align: justify; padding-bottom:6px;"><strong>Etcd</strong><br>
Settings specific to Etcd DCS.</p>
<p style="text-align: justify; padding-bottom:6px;"><strong>Exhibitor</strong><br>
Settings specific to Exhibitor DCS.</p>
<p style="text-align: justify; padding-bottom:6px;"><strong>Kubernetes</strong><br>
Settings specific to Kubernetes DCS.</p>
<p style="text-align: justify; padding-bottom:6px;"><strong>ZooKeeper</strong><br>
Settings specific to ZooKeeper DCS.</p>
<p style="text-align: justify; padding-bottom:6px;"><strong>Watchdog</strong><br>
Settings specific to Watchdog.</p></td>
</tr>
</tbody>
</table>
</li>
</ul>
<h2 style="padding-top: 15px;">Patroni Pros</h2>
<ul>
	<li>Patroni enables end-to-end setup up of the cluster.</li>
	<li>Supports REST APIs and HAproxy integration.</li>
	<li>Supports event notifications via callbacks scripts triggered by certain actions.</li>
	<li>Leverages DCS for consensus.</li>
</ul>
<h2 style="padding-top: 15px;">Patroni Cons</h2>
<ul>
	<li>Patroni will not detect the misconfiguration of a standby with an unknown or non-existent node in recovery configuration. The node will be shown as a slave even if the standby is running without connecting to the master/cascading standby node.</li>
	<li>User needs to handle setup, management, and upgrade of the DCS software.</li>
	<li>Requires multiple ports to be open for components communication:
<ul>
	<li>REST API port for Patroni</li>
	<li>Minimum 2 ports for DCS</li>
</ul>
</li>
</ul>
<h2 style="padding-top: 15px;">High Availability Test Scenarios</h2>
<p style="text-align: justify;">We conducted a few tests on PostgreSQL HA management using Patroni. All of these tests were performed while the application was running and inserting data to the PostgreSQL database. The application was written using PostgreSQL Java <a style="color: #2da964;" href="https://jdbc.postgresql.org/" target="_blank">JDBC Driver</a> leveraging the connection failover capability.</p>

<h3 style="padding-top: 15px;">Standby Server Tests</h3>
<table class="table table-bordered">
<tbody>
<tr>
<th style="width: 5%; vertical-align: middle; background-color: #d9fce9;">Sl. No</th>
<th style="width: 25%; vertical-align: middle; background-color: #d9fce9;">Test Scenario</th>
<th style="width: 70%; vertical-align: middle; background-color: #d9fce9;">Observation</th>
</tr>
<tr>
<td style="vertical-align: middle;">1</td>
<td style="vertical-align: middle; background-color: #f5f5f5;">Kill the PostgreSQL process</td>
<td style="vertical-align: middle;">Patroni brought the PostgreSQL process back to running state.
<ul>
	<li>There was no disruption of the writer application.</li>
</ul>
</td>
</tr>
<tr>
<td style="vertical-align: middle;">2</td>
<td style="vertical-align: middle; background-color: #f5f5f5;">Stop the PostgreSQL process</td>
<td style="vertical-align: middle;">Patroni brought the PostgreSQL process back to running state.
<ul>
	<li>There was no disruption of the writer application.</li>
</ul>
</td>
</tr>
<tr>
<td style="vertical-align: middle;">3</td>
<td style="vertical-align: middle; background-color: #f5f5f5;">Reboot the server</td>
<td style="vertical-align: middle;">Patroni needs to be started after reboot, unless configured to not start on reboot. Once Patroni was started, it started the PostgreSQL process and setup the standby configuration.
<ul>
	<li>There was no disruption of the writer application.</li>
</ul>
</td>
</tr>
<tr>
<td style="vertical-align: middle;">4</td>
<td style="vertical-align: middle; background-color: #f5f5f5;">Stop the Patroni process</td>
<td style="vertical-align: middle;">
<ul>
	<li>It did not stop the PostgreSQL process.</li>
	<li><em>patronictl list</em> did not display this server.</li>
	<li>There was no disruption of the writer application.</li>
</ul>
So, essentially, you need to monitor the health of the Patroni process - otherwise it will lead to issues down the line.</td>
</tr>
</tbody>
</table>
<h3 style="padding-top: 15px;">Master/Primary Server Tests</h3>
<table class="table table-bordered">
<tbody>
<tr>
<td style="width: 5%; vertical-align: middle; background-color: #d9fce9;"><strong>Sl. No</strong></td>
<td style="width: 25%; vertical-align: middle; background-color: #d9fce9;"><strong>Test Scenario</strong></td>
<td style="width: 70%; vertical-align: middle; background-color: #d9fce9;"><strong>Observation</strong></td>
</tr>
<tr>
<td style="vertical-align: middle;">1</td>
<td style="vertical-align: middle; background-color: #f5f5f5;">Kill the PostgreSQL process</td>
<td style="vertical-align: middle;">Patroni brought the PostgreSQL process back to running state. Patroni running on that node had primary lock and so the election was not triggered.
<ul>
	<li>There was downtime in the writer application.</li>
</ul>
</td>
</tr>
<tr>
<td style="vertical-align: middle;">2</td>
<td style="vertical-align: middle; background-color: #f5f5f5;">Stop the PostgreSQL process and bring it back immediately after health check expiry</td>
<td style="vertical-align: middle;">Patroni brought the PostgreSQL process back to running state. Patroni running on that node had primary lock and so the election was not triggered.
<ul>
	<li>There was downtime in the writer application.</li>
</ul>
</td>
</tr>
<tr>
<td style="vertical-align: middle;">3</td>
<td style="vertical-align: middle; background-color: #f5f5f5;">Reboot the server</td>
<td style="vertical-align: middle;">Failover happened and one of the standby servers was elected as the new master after obtaining the lock. When Patroni was started on the old master, it brought back the old master up and performed pg_rewind and started following the new master.T
<ul>
	<li>There was downtime in the writer application.</li>
</ul>
</td>
</tr>
<tr>
<td style="vertical-align: middle;">4</td>
<td style="vertical-align: middle; background-color: #f5f5f5;">Stop/Kill the Patroni process</td>
<td style="vertical-align: middle;">
<ul>
	<li>One of the standby servers acquired the DCS lock and became the master by promoting itself.</li>
	<li>The old master was still running and it led to multi-master scenario. The application was still writing to the old master.</li>
	<li>Once Patroni was started on the old master, it rewound the old master (<strong>use_pg_rewind</strong> was set to true) to the new master timeline and lsn and started following the new master.</li>
</ul>
As you can see above, it is very important to monitor the health of the Patroni process on the master. Failure to do so can lead to a multi-master scenario and potential data loss.</td>
</tr>
</tbody>
</table>
<h3 style="padding-top: 15px;">Network Isolation Tests</h3>
<table class="table table-bordered">
<tbody>
<tr>
<td style="width: 5%; vertical-align: middle; background-color: #d9fce9;"><strong>Sl. No</strong></td>
<td style="width: 25%; vertical-align: middle; background-color: #d9fce9;"><strong>Test Scenario</strong></td>
<td style="width: 70%; vertical-align: middle; background-color: #d9fce9;"><strong>Observation</strong></td>
</tr>
<tr>
<td style="vertical-align: middle;">1</td>
<td style="vertical-align: middle; background-color: #f5f5f5;">Network-isolate the master server from other servers</td>
<td style="vertical-align: middle;">DCS communication was blocked for master node.
<ul>
	<li>PostgreSQL was demoted on the master server.</li>
	<li>A new master was elected in the majority partition.</li>
	<li>There was a downtime in the writer application.</li>
</ul>
</td>
</tr>
<tr>
<td style="vertical-align: middle;">2</td>
<td style="vertical-align: middle; background-color: #f5f5f5;">Network-isolate the standby server from other servers</td>
<td style="vertical-align: middle;">DCS communication was blocked for the standby node.
<ul>
	<li>The PostgreSQL service was running, however, the node was not considered for elections.</li>
	<li>There was no disruption in the writer application.</li>
</ul>
</td>
</tr>
</tbody>
</table>
<h2 style="padding-top: 15px;">What's the Best PostgreSQL HA Framework?</h2>
<p style="text-align: justify;">Patroni is a valuable tool for PostgreSQL database administrators (DBAs), as it performs end-to-end setup and monitoring of a PostgreSQL cluster. The flexibility of choosing DCS and standby creation is an advantage to the end user, as they can choose the method they are comfortable with.</p>
<p style="text-align: justify;">REST APIs, HaProxy integration, Watchdog support, callbacks and its feature-rich management makes Patroni the best solution for PostgreSQL HA management.</p>

<h2 style="padding-top: 15px;">PostgreSQL HA Framework Testing: PAF vs. repmgr vs. Patroni</h2>
<p style="text-align: justify;">Included below is a comprehensive table detailing the results of all the tests we have performed on all three frameworks - PostgreSQL Automatic Failover (PAF), Replication Manager (repmgr) and Patroni.</p>

<h3 style="padding-top: 15px;">Standby Server Tests</h3>
<table class="table table-bordered">
<tbody>
<tr>
<th style="width: 13%; vertical-align: middle; background-color: #d9fce9;">Test Scenario</th>
<th style="width: 29%; vertical-align: middle; background-color: #d9fce9;">PostgreSQL Automatic Failover (PAF)</th>
<th style="width: 29%; vertical-align: middle; background-color: #d9fce9;">Replication Manager (repmgr)</th>
<th style="width: 29%; vertical-align: middle; background-color: #d9fce9;">Patroni</th>
</tr>
<tr>
<td style="vertical-align: middle; background-color: #f5f5f5;">Kill the PostgreSQL process</td>
<td style="vertical-align: middle; font-size: 15px;">Pacemaker brought the PostgreSQL process back to running state.
<ul>
	<li>There was no disruption of the writer application.</li>
</ul>
</td>
<td style="vertical-align: middle; font-size: 15px;">Standby server was marked as failed. Manual intervention was required to start the PostgreSQL process again.
<ul>
	<li>There was no disruption of the writer application.</li>
</ul>
</td>
<td style="vertical-align: middle; font-size: 15px;">Patroni brought the PostgreSQL process back to running state.
<ul>
	<li>There was no disruption of the writer application.</li>
</ul>
</td>
</tr>
<tr>
<td style="vertical-align: middle; background-color: #f5f5f5;">Stop the PostgreSQL process</td>
<td style="vertical-align: middle; font-size: 15px;">Pacemaker brought the PostgreSQL process back to running state.
<ul>
	<li>There was no disruption of the writer application.</li>
</ul>
</td>
<td style="vertical-align: middle; font-size: 15px;">Standby server was marked as failed. Manual intervention was required to start the PostgreSQL process again.
<ul>
	<li>There was no disruption of the writer application.</li>
</ul>
</td>
<td style="vertical-align: middle; font-size: 15px;">Patroni brought the PostgreSQL process back to running state.
<ul>
	<li>There was no disruption of the writer application.</li>
</ul>
</td>
</tr>
<tr>
<td style="vertical-align: middle; background-color: #f5f5f5;">Reboot the server</td>
<td style="vertical-align: middle; font-size: 15px;">Standby server was marked offline initially. Once the server came up after reboot, PostgreSQL was started by Pacemaker and the server was marked as online. If fencing was enabled then node wouldn’t have been added automatically to cluster.
<ul>
	<li>There was no disruption of the writer application.</li>
</ul>
</td>
<td style="vertical-align: middle; font-size: 15px;">Standby server was marked as failed. Once the server came up after reboot, PostgreSQL was started manually and server was marked as running.
<ul>
	<li>There was no disruption of the writer application.</li>
</ul>
</td>
<td style="vertical-align: middle; font-size: 15px;">Patroni needs to be started after reboot, unless configured to not start on reboot. Once Patroni was started, it started the PostgreSQL process and setup the standby configuration.
<ul>
	<li>There was no disruption of the writer application.</li>
</ul>
</td>
</tr>
<tr>
<td style="vertical-align: middle; background-color: #f5f5f5;">Stop the framework agent process</td>
<td style="vertical-align: middle; font-size: 15px;">Agent: pacemaker
<ul>
	<li>The PostgreSQL process was stopped and was marked offline.</li>
	<li>There was no disruption of the writer application.</li>
</ul>
</td>
<td style="vertical-align: middle; font-size: 15px;">Agent: repmgrd
<ul>
	<li>The standby server will not be part of automated failover situation.</li>
	<li>PostgreSQL service was found to be running.</li>
	<li>There was no disruption of the writer application.</li>
</ul>
</td>
<td style="vertical-align: middle; font-size: 15px;">Agent: patroni
<ul>
	<li>It did not stop the PostgreSQL process.</li>
	<li><em>patronictl list</em> did not display this server.</li>
	<li>There was no disruption of the writer application.</li>
</ul>
</td>
</tr>
</tbody>
</table>
<h3 style="padding-top: 15px;">Master/Primary Server Tests</h3>
<table class="table table-bordered">
<tbody>
<tr>
<th style="width: 13%; vertical-align: middle; background-color: #d9fce9;">Test Scenario</th>
<th style="width: 29%; vertical-align: middle; background-color: #d9fce9;">PostgreSQL Automatic Failover (PAF)</th>
<th style="width: 29%; vertical-align: middle; background-color: #d9fce9;">Replication Manager (repmgr)</th>
<th style="width: 29%; vertical-align: middle; background-color: #d9fce9;">Patroni</th>
</tr>
<tr>
<td style="vertical-align: middle; background-color: #f5f5f5;">Kill the PostgreSQL process</td>
<td style="vertical-align: middle; font-size: 15px;">Pacemaker brought the PostgreSQL process back to running state. Primary got recovered within the threshold time and hence election was not triggered.
<ul>
	<li>There was downtime in the writer application.</li>
</ul>
</td>
<td style="vertical-align: middle; font-size: 15px;">repmgrd started health check for primary server connection on all standby servers for a fixed interval. When all retries failed, an election was triggered on all the standby servers. As a result of the election, the standby which had the latest received LSN got promoted. The standby servers which lost the election will wait for the notification from the new master node and will follow it once they receive the notification.Manual intervention was required to start the postgreSQL process again.
<ul>
	<li>There was downtime in the writer application.</li>
</ul>
</td>
<td style="vertical-align: middle; font-size: 15px;">Patroni brought the PostgreSQL process back to running state. Patroni running on that node had primary lock and hence election was not triggered.
<ul>
	<li>There was downtime in the writer application.</li>
</ul>
</td>
</tr>
<tr>
<td style="vertical-align: middle; background-color: #f5f5f5;">Stop the PostgreSQL process and bring it back immediately after health check expiry</td>
<td style="vertical-align: middle; font-size: 15px;">Pacemaker brought the PostgreSQL process back to running state. Primary got recovered within the threshold time and hence election was not triggered.
<ul>
	<li>There was downtime in the writer application.</li>
</ul>
</td>
<td style="vertical-align: middle; font-size: 15px;">repmgrd started health check for primary server connections on all standby servers for a fixed interval. When all the retries failed, an election was triggered on all the standby nodes. However, the newly elected master didn’t notify the existing standby servers since the old master was back.Cluster was left in an indeterminate state and manual intervention was required.
<ul>
	<li>There was downtime in the writer application.</li>
</ul>
</td>
<td style="vertical-align: middle; font-size: 15px;">Patroni brought the PostgreSQL process back to running state. Patroni running on that node had primary lock and hence election was not triggered.
<ul>
	<li>There was downtime in the writer application.</li>
</ul>
</td>
</tr>
<tr>
<td style="vertical-align: middle; background-color: #f5f5f5;">Reboot the server</td>
<td style="vertical-align: middle; font-size: 15px;">Election was triggered by Pacemaker after the threshold time for which master was not available. The most eligible standby server was promoted as the new master. Once the old master came up after reboot, it was added back to the cluster as a standby. If fencing was enabled, then node wouldn’t have been added automatically to cluster.
<ul>
	<li>There was downtime in the writer application.</li>
</ul>
</td>
<td style="vertical-align: middle; font-size: 15px;">repmgrd started election when master connection health check failed on all standby servers. The eligible standby was promoted. When this server came back, it didn’t join the cluster and was marked failed. repmgr node rejoin command was run to add the server back to the cluster.
<ul>
	<li>There was downtime in the writer application.</li>
</ul>
</td>
<td style="vertical-align: middle; font-size: 15px;">Failover happened and one of the standby servers was elected as the new master after obtaining the lock. When Patroni was started on the old master, it brought back the old master up and performed pg_rewind and started following the new master.
<ul>
	<li>There was downtime in the writer application.</li>
</ul>
</td>
</tr>
<tr>
<td style="vertical-align: middle; background-color: #f5f5f5;">Stop the framework agent process</td>
<td style="vertical-align: middle; font-size: 15px;">Agent: pacemaker
<ul>
	<li>The PostgreSQL process was stopped and it was marked offline.</li>
	<li>Election was triggered and new master was elected.</li>
	<li>There was downtime in writer application.</li>
</ul>
</td>
<td style="vertical-align: middle; font-size: 15px;">Agent: repmgrd
<ul>
	<li>The primary server will not be part of the automated failover situation.</li>
	<li>PostgreSQL service was found to be running.</li>
	<li>There was no disruption in writer application.</li>
</ul>
</td>
<td style="vertical-align: middle; font-size: 15px;">Agent: patroni
<ul>
	<li>One of the standby servers acquired the DCS lock and became the master by promoting itself.</li>
	<li>The old master was still running and it led to multi-master scenario. The application was still writing to the old master.</li>
	<li>Once Patroni was started on the old master, it rewound the old master (<strong>use_pg_rewind</strong> was set to true) to the new master timeline and lsn and started following the new master.</li>
</ul>
</td>
</tr>
</tbody>
</table>
<h3 style="padding-top: 15px;">Network Isolation Tests</h3>
<table class="table table-bordered">
<tbody>
<tr>
<th style="width: 13%; vertical-align: middle; background-color: #d9fce9;">Test Scenario</th>
<th style="width: 29%; vertical-align: middle; background-color: #d9fce9;">PostgreSQL Automatic Failover (PAF)</th>
<th style="width: 29%; vertical-align: middle; background-color: #d9fce9;">Replication Manager (repmgr)</th>
<th style="width: 29%; vertical-align: middle; background-color: #d9fce9;">Patroni</th>
</tr>
<tr>
<td style="vertical-align: middle; background-color: #f5f5f5;">Network isolate the master server from other servers (split brain scenario)</td>
<td style="vertical-align: middle; font-size: 15px;">Corosync traffic was blocked on the master server.
<ul>
	<li>PostgreSQL service was turned off and master server was marked offline due to quorum policy.</li>
	<li>A new master was elected in the majority partition.</li>
	<li>There was a downtime in the writer application.</li>
</ul>
</td>
<td style="vertical-align: middle; font-size: 15px;">All servers have the same value for <em>location</em> in repmgr configuration:
<ul>
	<li>repmgrd started election when master connection health check failed on all standby servers.</li>
	<li>The eligible standby was promoted, but the PostgreSQL process was still running on the old master node.</li>
	<li>There were two nodes running as master. Manual intervention was required after the network isolation was corrected.</li>
</ul>
The standby servers have the same value for <em>location</em> but the primary had a different value for <em>location</em> in repmgr configuration:
<ul>
	<li>repmgrd started election when master connection health check failed on all standby servers.</li>
	<li>But, there was no new master elected since the standby servers had <em>location</em> different from that of the primary.</li>
	<li>repmgrd went into degrade monitoring mode. PostgreSQL was running on all the nodes and there was only one master in the cluster.</li>
</ul>
</td>
<td style="vertical-align: middle; font-size: 15px;">DCS communication was blocked for master node.
<ul>
	<li>PostgreSQL was demoted on the master server.</li>
	<li>A new master was elected in the majority partition.</li>
	<li>There was a downtime in the writer application.</li>
</ul>
</td>
</tr>
<tr>
<td style="vertical-align: middle; background-color: #f5f5f5;">Network-isolate the standby server from other servers</td>
<td style="vertical-align: middle; font-size: 15px;">Corosync traffic was blocked on the standby server.
<ul>
	<li>The server was marked offline and PostgreSQL service was turned off due to quorum policy.</li>
	<li>There was no disruption in the writer application.</li>
</ul>
</td>
<td style="vertical-align: middle; font-size: 15px;">
<ul>
	<li>repmgrd went into degrade monitoring mode.</li>
	<li>The PostgreSQL process was still running on the standby node.</li>
	<li>Manual intervention was required after the network isolation was corrected.</li>
</ul>
</td>
<td style="vertical-align: middle; font-size: 15px;">DCS communication was blocked for the standby node.
<ul>
	<li>The PostgreSQL service was running, however, the node was not considered for elections.</li>
	<li>There was no disruption in the writer application.</li>
</ul>
</td>
</tr>
</tbody>
</table>