PyMongo Tutorial: Testing MongoDB Failover in Your Python App
Tags: mongodb, python, database, highavailability
Published: 2019-05-02 16:05:27

---
title: PyMongo Tutorial: Testing MongoDB Failover in Your Python App
published: true
description: How to connect an SSL-enabled MongoDB replica set with self-signed certificates using PyMongo, and test MongoDB failover behavior in Python code: ScaleGrid Blog
tags: MongoDB, Python, Database, High Availability,
---
<p style="padding-bottom: 10px;"><a href="https://scalegrid.io/blog/pymongo-tutorial-testing-mongodb-failover-in-your-python-app/"><img class="aligncenter size-full wp-image-3734" src="https://scalegrid.io/blog/wp-content/uploads/2019/05/PyMongo-Tutorial-Testing-MongoDB-Failovers-for-Python-Applications-ScaleGrid-Blog.jpg" alt="PyMongo Tutorial: Testing MongoDB Failover in Your Python App" width="100%" height="auto" /></a>
<p style="text-align: justify;">Python is a powerful and flexible programming language used by millions of developers around the world to build their applications. It comes as no surprise that Python developers commonly leverage <a style="color: #2da964;" title="MongoDB Hosting" href="https://scalegrid.io/mongodb.html" target="_blank">MongoDB hosting</a>, the <a style="color: #2da964;" href="https://scalegrid.io/blog/2019-database-trends-sql-vs-nosql-top-databases-single-vs-multiple-database-use/" target="_blank">most popular NoSQL database</a>, for their deployments due to its flexible nature and lack of schema requirements.</p>
<p style="text-align: justify;">So, what's the best way to use MongoDB with Python? <a style="color: #2da964;" href="https://api.mongodb.com/python/current/" target="_blank" rel="nofollow">PyMongo</a> is a Python distribution containing tools for working with MongoDB, and the recommended <a style="color: #2da964;" href="https://docs.mongodb.com/ecosystem/drivers/python/" target="_blank" rel="nofollow">Python MongoDB</a> driver. It is a fairly mature driver that supports <a style="color: #2da964;" href="https://api.mongodb.com/python/current/examples/index.html" target="_blank" rel="nofollow">most of the common</a> operations with the database, and you can check out this <a style="color: #2da964;" href="https://api.mongodb.com/python/current/tutorial.html" target="_blank" rel="nofollow">tutorial</a> for an introduction to the PyMongo driver.</p>
<p style="text-align: justify;">When deploying in production, it's highly recommended to setup in a <a style="color: #2da964;" href="https://scalegrid.io/blog/geo-distributed-mongodb-replica-sets-for-100-uptime/" target="_blank">MongoDB replica set configuration</a> so your data is geographically distributed for high availability. It is also recommended that SSL connections be enabled to encrypt the client-database traffic. We often undertake testing of <a style="color: #2da964;" href="https://docs.mongodb.com/manual/reference/glossary/#term-failover" target="_blank" rel="nofollow">failover </a>characteristics of various MongoDB drivers to qualify them for production use cases, or when our customers ask us for advice. In this post, we show you how to connect to an SSL-enabled MongoDB replica set configured with self-signed certificates using PyMongo, and how to <a style="color: #2da964;" href="https://scalegrid.io/blog/pymongo-tutorial-testing-mongodb-failover-in-your-python-app/" target="_blank">test MongoDB failover behavior in your code</a>.</p>

<h2>Connecting to MongoDB SSL Using Self-Signed Certificates</h2>
<p style="text-align: justify;">The first step is to ensure that the right versions of PyMongo and its dependencies are installed. <a style="color: #2da964;" href="https://api.mongodb.com/python/current/examples/tls.html#dependencies" target="_blank" rel="nofollow">This guide</a> helps you in sorting out the dependencies, and the driver compatibility matrix can be found <a style="color: #2da964;" href="https://docs.mongodb.com/ecosystem/drivers/python/#python-driver-compatibility" target="_blank" rel="nofollow">here</a>.</p>
<p style="text-align: justify;">The <a style="color: #2da964;" href="https://api.mongodb.com/python/current/api/pymongo/mongo_client.html#pymongo.mongo_client.MongoClient" target="_blank" rel="nofollow"><tt>mongo_client.MongoClient</tt></a> parameters that are of interest to us are <strong><tt>ssl</tt></strong> and <strong><tt>ss_ca_cert</tt></strong>. In order to connect to an SSL-enabled MongoDB endpoint that uses a self-signed certificate, <strong><tt>ssl</tt></strong> must be set to <strong><tt>True</tt></strong> and <strong><tt>ss_ca_cert</tt></strong> must point to the <a style="color: #2da964;" href="https://en.wikipedia.org/wiki/Certificate_authority" target="_blank">CA</a> certificate file.</p>
<p style="text-align: justify;">If you are a <a style="color: #2da964;" href="https://scalegrid.io" target="_blank">ScaleGrid</a> customer, you can <a style="color: #2da964;" href="https://help.scalegrid.io/docs/mongodb-connecting-to" target="_blank">download the CA certificate file for your MongoDB clusters</a> from the <a style="color: #2da964;" href="https://console.scalegrid.io" target="_blank">ScaleGrid console</a> as shown here:</p>
<p style="text-align: justify; padding-bottom: 10px;"><a href="https://scalegrid.io/blog/wp-content/uploads/2018/12/SSLCert.jpg"><img class="aligncenter size-full wp-image-3734" style="padding: 10px 0px 15px 0px;" src="https://scalegrid.io/blog/wp-content/uploads/2018/12/SSLCert.jpg" alt="Download the CA Certificate File for Your MongoDB Clusters From the ScaleGrid Console" width="100%" height="auto" /></a>So, a connection snippet would look like:</p>

<pre>&gt;&gt;&gt; import pymongo
&gt;&gt;&gt; MONGO_URI = 'mongodb://rwuser:@SG-example-0.servers.mongodirector.com:27017,SG-example-1.servers.mongodirector.com:27017,SG-example-2.servers.mongodirector.com:27017/admin?replicaSet=RS-example&amp;ssl=true'
&gt;&gt;&gt; client = pymongo.MongoClient(MONGO_URI, ssl = True, ssl_ca_certs = '')
&gt;&gt;&gt; print("Databases - " + str(client.list_database_names()))
Databases - ['admin', 'local', 'test']
&gt;&gt;&gt; client.close()
&gt;&gt;&gt;
</pre>
<p style="text-align: justify; padding-top: 10px;">If you are <a style="color: #2da964;" title="MongoDB – Bring Your Own SSL Certificates" href="https://scalegrid.io/blog/bring-your-own-ssl-certificates/" target="_blank">using your own self-signed certificates</a> where <a style="color: #2da964;" href="https://wiki.openssl.org/index.php/Hostname_validation" target="_blank">hostname verification</a> might fail, you will also have to set the <strong><tt>ssl_match_hostname</tt></strong> parameter to <strong><tt>False</tt></strong>. Like the driver documentation says, this is not recommended as it makes the connection susceptible to <a style="color: #2da964;" href="https://en.wikipedia.org/wiki/Man-in-the-middle_attack" target="_blank">man-in-the-middle attacks</a>.</p>

<h2>Testing Failover Behavior</h2>
<p style="text-align: justify;">With MongoDB deployments, failovers aren't considered major events as they were with traditional database management systems. Although most MongoDB drivers try to abstract this event, developers should understand and design their applications for such behavior, as applications should expect transient network errors and retry before percolating errors up.</p>
<p style="text-align: justify; padding-bottom: 10px;">You can test the resilience of your applications by inducing failovers while your workload runs. The easiest way to induce failover is to run the <a style="color: #2da964;" href="https://docs.mongodb.com/manual/reference/method/rs.stepDown/" target="_blank" rel="nofollow">rs.stepDown()</a> command:</p>

<pre>RS-example-0:PRIMARY&gt; rs.stepDown()
2019-04-18T19:44:42.257+0530 E QUERY [thread1] Error: error doing query: failed: network error while attempting to run command 'replSetStepDown' on host 'SG-example-1.servers.mongodirector.com:27017' :
DB.prototype.runCommand@src/mongo/shell/db.js:168:1
DB.prototype.adminCommand@src/mongo/shell/db.js:185:1
rs.stepDown@src/mongo/shell/utils.js:1305:12
@(shell):1:1
2019-04-18T19:44:42.261+0530 I NETWORK [thread1] trying reconnect to SG-example-1.servers.mongodirector.com:27017 (X.X.X.X) failed
2019-04-18T19:44:43.267+0530 I NETWORK [thread1] reconnect SG-example-1.servers.mongodirector.com:27017 (X.X.X.X) ok
RS-example-0:SECONDARY&gt;
</pre>
<p style="text-align: justify; padding: 10px 0px;">One of the ways I like to test the behavior of drivers is by writing a simple 'perpetual' writer app. This would be simple code that keeps writing to the database unless interrupted by the user, and would print all exceptions it encounters to help us understand the driver and database behavior. I also keep track of the data it writes to ensure that there's no unreported data loss in the test. Here's the relevant part of test code we will use to test our MongoDB failover behavior:</p>

<pre>import logging
import traceback
...
import pymongo
...
logger = logging.getLogger("test")

MONGO_URI = 'mongodb://rwuser:@SG-example-0.servers.mongodirector.com:48273,SG-example-1.servers.mongodirector.com:27017,SG-example-2.servers.mongodirector.com:27017/admin?replicaSet=RS-example-0&amp;ssl=true'

try:
    logger.info("Attempting to connect...")
    client = pymongo.MongoClient(MONGO_URI, ssl = True, ssl_ca_certs = 'path-to-cacert.pem')
    db = client['test']
    collection = db['test']
    i = 0
    while True:
        try:
            text = ''.join(random.choices(string.ascii_uppercase + string.digits, k = 3))
            doc = { "idx": i, "date" : datetime.utcnow(), "text" : text}
            i += 1
            id = collection.insert_one(doc).inserted_id
            logger.info("Record inserted - id: " + str(id))
            sleep(3)
        except pymongo.errors.ConnectionFailure as e:
            logger.error("ConnectionFailure seen: " + str(e))
            traceback.print_exc(file = sys.stdout)
            logger.info("Retrying...")

    logger.info("Done...")
except Exception as e:
    logger.error("Exception seen: " + str(e))
    traceback.print_exc(file = sys.stdout)
finally:
    client.close()

</pre>
<p style="text-align: justify; padding: 10px 0px;">The sort of entries that this writes look like:</p>

<pre>RS-example-0:PRIMARY&gt; db.test.find()
{ "_id" : ObjectId("5cb6d6269ece140f18d05438"), "idx" : 0, "date" : ISODate("2019-04-17T07:30:46.533Z"), "text" : "400" }
{ "_id" : ObjectId("5cb6d6299ece140f18d05439"), "idx" : 1, "date" : ISODate("2019-04-17T07:30:49.755Z"), "text" : "X63" }
{ "_id" : ObjectId("5cb6d62c9ece140f18d0543a"), "idx" : 2, "date" : ISODate("2019-04-17T07:30:52.976Z"), "text" : "5BX" }
{ "_id" : ObjectId("5cb6d6329ece140f18d0543c"), "idx" : 4, "date" : ISODate("2019-04-17T07:30:58.001Z"), "text" : "TGQ" }
{ "_id" : ObjectId("5cb6d63f9ece140f18d0543d"), "idx" : 5, "date" : ISODate("2019-04-17T07:31:11.417Z"), "text" : "ZWA" }
{ "_id" : ObjectId("5cb6d6429ece140f18d0543e"), "idx" : 6, "date" : ISODate("2019-04-17T07:31:14.654Z"), "text" : "WSR" }
..
</pre>
<h3 style="padding-top: 10px;">Handling the ConnectionFailure Exception</h3>
<p style="text-align: justify;">Notice that we catch the <a style="color: #2da964;" href="https://api.mongodb.com/python/current/api/pymongo/errors.html#pymongo.errors.ConnectionFailure" target="_blank" rel="nofollow">ConnectionFailure</a> exception to deal with all network-related issues we may encounter due to failovers - we print the exception and continue to attempt to write to the database. The driver documentation <a style="color: #2da964;" href="https://api.mongodb.com/python/current/api/pymongo/mongo_client.html#pymongo.mongo_client.MongoClient" target="_blank" rel="nofollow">recommends </a>that:</p>

<blockquote>If an operation fails because of a network error, ConnectionFailure is raised and the client reconnects in the background. Application code should handle this exception (recognizing that the operation failed) and then continue to execute.</blockquote>
<p style="text-align: justify; padding-bottom: 10px;">Let's run this and do a database failover while it executes. Here's what happens:</p>

<pre>04/17/2019 12:49:17 PM INFO Attempting to connect...
04/17/2019 12:49:20 PM INFO Record inserted - id: 5cb6d3789ece145a2408cbc7
04/17/2019 12:49:23 PM INFO Record inserted - id: 5cb6d37b9ece145a2408cbc8
04/17/2019 12:49:27 PM INFO Record inserted - id: 5cb6d37e9ece145a2408cbc9
04/17/2019 12:49:30 PM ERROR PyMongoError seen: connection closed
Traceback (most recent call last):
    id = collection.insert_one(doc).inserted_id
  File "C:\Users\Random\AppData\Local\Programs\Python\Python36-32\lib\site-packages\pymongo\collection.py", line 693, in insert_one
    session=session),
...
  File "C:\Users\Random\AppData\Local\Programs\Python\Python36-32\lib\site-packages\pymongo\network.py", line 173, in receive_message
    _receive_data_on_socket(sock, 16))
  File "C:\Users\Random\AppData\Local\Programs\Python\Python36-32\lib\site-packages\pymongo\network.py", line 238, in _receive_data_on_socket
    raise AutoReconnect("connection closed")
pymongo.errors.AutoReconnect: connection closed
<strong>04/17/2019 12:49:30 PM INFO Retrying...
04/17/2019 12:49:42 PM INFO Record inserted - id: 5cb6d3829ece145a2408cbcb</strong>
04/17/2019 12:49:45 PM INFO Record inserted - id: 5cb6d3919ece145a2408cbcc
04/17/2019 12:49:49 PM INFO Record inserted - id: 5cb6d3949ece145a2408cbcd
04/17/2019 12:49:52 PM INFO Record inserted - id: 5cb6d3989ece145a2408cbce
</pre>
<p style="text-align: justify; padding-top: 10px;">Notice that the driver takes about 12 seconds to understand the new topology, connect to the new primary, and continue writing.  The exception raised is <a style="color: #2da964;" href="https://api.mongodb.com/python/current/api/pymongo/errors.html#pymongo.errors.AutoReconnect" target="_blank" rel="nofollow"><tt>errors.AutoReconnect</tt></a> which is a subclass of <tt>ConnectionFailure</tt>.</p>

<p style="text-align: justify; padding-bottom: 10px;">You could do a few more runs to see what other exceptions are seen. For example, here's another exception trace I encountered:</p>

<pre>    id = collection.insert_one(doc).inserted_id
  File "C:\Users\Random\AppData\Local\Programs\Python\Python36-32\lib\site-packages\pymongo\collection.py", line 693, in insert_one
    session=session),
...
  File "C:\Users\Randome\AppData\Local\Programs\Python\Python36-32\lib\site-packages\pymongo\network.py", line 150, in command
    parse_write_concern_error=parse_write_concern_error)
  File "C:\Users\Random\AppData\Local\Programs\Python\Python36-32\lib\site-packages\pymongo\helpers.py", line 132, in _check_command_response
    raise NotMasterError(errmsg, response)
pymongo.errors.NotMasterError: not master
</pre>
<p style="text-align: justify; padding-top: 10px;">This exception is also a sub class of <tt>ConnectionFailure.</tt></p>

<h3>'retryWrites' Parameter</h3>
<p style="text-align: justify;">Another area to test MongoDB failover behavior would be seeing how other parameter variations affect the results. One parameter that is relevant is '<tt><strong>retryWrites</strong></tt>':</p>

<blockquote>retryWrites: (boolean) Whether supported write operations executed within this MongoClient will be retried once after a network error on MongoDB 3.6+. Defaults to False.</blockquote>
<p style="text-align: justify; padding-bottom: 10px;">Let's see how this parameter works with a failover. The only change made to the code is:</p>

<pre>client = pymongo.MongoClient(MONGO_URI, ssl = True, ssl_ca_certs = 'path-to-cacert.pem', retryWrites = True)
</pre>
<p style="text-align: justify; padding: 10px 0px;">Let's run it now, and then do a database system failover:</p>

<pre>04/18/2019 08:49:30 PM INFO Attempting to connect...
04/18/2019 08:49:35 PM INFO Record inserted - id: 5cb895869ece146554010c77
04/18/2019 08:49:38 PM INFO Record inserted - id: 5cb8958a9ece146554010c78
04/18/2019 08:49:41 PM INFO Record inserted - id: 5cb8958d9ece146554010c79
04/18/2019 08:49:44 PM INFO Record inserted - id: 5cb895909ece146554010c7a
<strong>04/18/2019 08:49:48 PM INFO Record inserted - id: 5cb895939ece146554010c7b &lt;&lt;&lt; Failover around this time
04/18/2019 08:50:04 PM INFO Record inserted - id: 5cb895979ece146554010c7c</strong>
04/18/2019 08:50:07 PM INFO Record inserted - id: 5cb895a79ece146554010c7d
04/18/2019 08:50:10 PM INFO Record inserted - id: 5cb895aa9ece146554010c7e
04/18/2019 08:50:14 PM INFO Record inserted - id: 5cb895ad9ece146554010c7f
...
</pre>
<p style="text-align: justify; padding-top: 10px;">Notice how the insert after the failover takes about 12 seconds, but goes through successfully as the <strong><tt>retryWrites</tt></strong> parameter ensures the failed write is retried. Remember that setting this parameter doesn't absolve you from handling the <tt>ConnectionFailure</tt> exception - you need to worry about reads and other operations whose behavior is not affected by this parameter. It also doesn't completely solve the issue, even for supported operations - sometimes failovers can take longer to complete and <strong><tt>retryWrites</tt></strong> alone will not be enough.</p>

<h3>Configuring the Network Timeout Values</h3>
<p style="text-align: justify;"><strong><tt>rs.stepDown()</tt></strong> induces a rather quick failover, as the replica set primary is intructed to become a secondary, and the secondaries hold an election to determine the new primary. In production deployments, network load, partition, and other such issues delay the detection of unavailability of the primary server, thus, prolonging your failover time. You would also often run into PyMongo errors like <a style="color: #2da964;" href="https://api.mongodb.com/python/current/api/pymongo/errors.html#pymongo.errors.ServerSelectionTimeoutError" target="_blank" rel="nofollow"><tt>errors.ServerSelectionTimeoutError</tt></a>, <tt>errors.NetworkTimeout,</tt> etc. during network issues and failovers.</p>
<p style="text-align: justify;">If this occurs very often, you must look to tweak the timeout parameters. The related <tt>MongoClient</tt> timeout parameters are <tt>serverSelectionTimeoutMS</tt>, <tt>connectTimeoutMS,</tt> and <tt>socketTimeoutMS</tt>. Of these, selecting a larger value for <tt>serverSelectionTimeoutMS</tt> most often helps in dealing with errors during failovers:</p>

<blockquote>serverSelectionTimeoutMS: (integer) Controls how long (in milliseconds) the driver will wait to find an available, appropriate server to carry out a database operation; while it is waiting, multiple server monitoring operations may be carried out, each controlled by connectTimeoutMS. Defaults to 30000 (30 seconds).</blockquote>
<p style="text-align: justify; padding-bottom: 15px;">Ready to use MongoDB in your Python application? Check out our <a style="color: #2da964;" href="https://scalegrid.io/blog/getting-started-with-python-and-mongodb-2/" target="_blank">Getting Started with Python and MongoDB</a> article to see how you can get up and running in just 5 easy steps. ScaleGrid is the only <a style="color: #2da964;" href="https://scalegrid.io/mongodb/hosting-comparison.html" target="_blank">MongoDB DBaaS provider</a> that it gives you full SSH access to your instances so you can run your Python server on the same machine as your MongoDB server. Automate your MongoDB cloud deployments on <a style="color: #2da964;" title="MongoDB hosting on AWS" href="https://scalegrid.io/mongodb/aws.html" target="_blank">AWS</a>, <a style="color: #2da964;" title="MongoDB hosting on Azure" href="https://scalegrid.io/mongodb/azure.html" target="_blank">Azure</a>, or <a style="color: #2da964;" title="MongoDB hosting on DigitalOcean" href="https://scalegrid.io/mongodb/digitalocean.html" target="_blank">DigitalOcean</a> with dedicated servers, high availability, and disaster recovery so you can focus on developing your Python application.</p>