How to Stop a Runaway Index Build in MongoDB
Tags: developer, index, mongodb, database
Published: 2017-09-21 15:52:36

---
title: How to Stop a Runaway Index Build in MongoDB
published: true
description: Experienced unresponsive clusters when triggering index builds on a large collection? These can last several hours or even days until the index build is complete. 
tags: Developer, Index, MongoDB, Database,
canonical_url: https://scalegrid.io/blog/how-to-stop-a-runaway-index-build-in-mongodb/ 
---
<link rel="canonical" href="https://scalegrid.io/blog/how-to-stop-a-runaway-index-build-in-mongodb/" />

<p><img src="https://cdn2.hubspot.net/hubfs/3848622/How-to-stop-a-runaway-index-build-in-MongoDB---ScaleGrid-Blog.jpg" alt="How to Stop a Runaway Index Build in MongoDB - ScaleGrid Blog" width="100%" height="auto" /></p>
<br>
<p style="text-align: justify;">Index builds in MongoDB can have an adverse impact on the availability of your MongoDB cluster. Â If you trigger a foreground index build on a large collection on your production server, you may find that your cluster is unresponsive until the index build is complete. Â On a large collection, this could take several hours or days, as described in <a title="The perils of index building in MongoDB - ScaleGrid Blog" href="https://scalegrid.io/blog/the-perils-of-building-indexes-on-mongodb/" target="_blank">the perils of index building in MongoDB</a>.</p>
<p style="text-align: justify;">The <a title="MongoDB Index Build Recommendation - ScaleGrid Blog" href="https://docs.mongodb.com/manual/core/index-creation/" target="_blank">recommended</a> best practice is to trigger index builds in the background, however, on large collection indexes, we've seen multiple problems with this approach. Â In the case of a three node cluster, both secondaries start building the index and stop responding to any requests. Consequently, the primary does not haveÂ quorum and moves to the secondary state taking your cluster down. Also, the default index builds triggered from the command line are foreground index builds - making this a now widespread problem. In future releases, we're hopefulÂ that this becomes background by default.</p>
<p style="text-align: justify;">Once you've triggered an index, simply restarting the server does not solve our problem; MongoDB will pick up the index build from where it left off. If you were running a background index build previously after theÂ restart, it now becomes a foreground index build, so in this case, the restart could make the problem worse.</p>
If you've already triggered an index build, how do you stop it? Luckily, it's relatively easy to stop an index build.
<h2>Option 1: Kill the index build process</h2>
Locate the index build process using db.currentOp() and then kill the operation using db.killOp(&lt;opid&gt;). The index operation will look something like this:

<table style="background:#f2f2f2;">
<tbody>
<tr>
<td>
[code language="javascript"]<br>
{<br>
"opid" : 820659355,<br>
"active" : true,<br>
"lockType" : "write",<br>
....<br>
"op" : "insert",<br>
"ns" : "xxxx",<br>
"query" : {<br>
},<br>
"client" : "xxxx",<br>
"desc" : "conn",<br>
"msg" : "index: (2/3) btree bottom up 292168587/398486401 64%"<br>
}<br>
[/code]<br>
</td>
</tr>
</tbody>
</table>

If the node where the index is building does not respond to new connections, or the killOp does not work, use Option 2 below:
<h2>Option 2: Configuring "noIndexBuildRetry" &amp; restart</h2>
<p style="text-align: justify;">MongoDB provides a <a title="noIndexBuildRetry - ScaleGrid Blog" href="https://docs.mongodb.com/v3.2/reference/program/mongod/" target="_blank">"--noIndexBuildRetry" </a>option which instructs MongoDB to stop building incomplete indexes on restart.</p>
This parameter doesn't appear to be supported from the config file, only as a parameter for the mongod process. We don't prefer to runÂ mongod manually with this optionÂ because if you accidentally run the mongod process as an elevated user (E.g. root), it ends up changing the permissions of all the files. Also, once run as "root", we've had intermittent problems running the process as mongod again.
<p style="text-align: justify;">A simpler option is to edit the /etc/init.d/mongod file. Looks for this line:</p>

<table style="background:#f2f2f2;">
<tbody>
<tr>
<td>
[code language="javascript"]<br>
OPTIONS=" -f $CONFIGFILE"<br>
[/code]<br>
</td>
</tr>
</tbody>
</table>


Replace with this line:

<table style="background:#f2f2f2;">
<tbody>
<tr>
<td>
[code language="javascript"]<br>
OPTIONS=" -f $CONFIGFILE --noIndexBuildRetry"<br>
[/code]<br>
</td>
</tr>
</tbody>
</table>

<h2>Detailed steps</h2>
For the purposes of this discussion, we're providing instructions for CentOS/RedHat/Amazon Linux.
<ol>
	<li>
<h3>Configure "--noIndexBuildRetry"</h3>
Add the "--noIndexBuildRetry" option to all your data nodes as explained above.</li>
	<li>
<h3>Restart all the nodes building the index</h3>
Look at the mongod log file for each data server and determine if it's building the index. If it is, restart the server "service mongod restart".</li>
	<li>
<h3>Drop the incomplete index</h3>
Once all of the relevant nodes are restarted, look at the list of indexes and drop the incomplete index if you see it on the list.</li>
	<li>
<h3>Remove "--noIndexBuildRetry"</h3>
Edit the /etc/init.d/mongod file to remove the --noIndexBuildRetry option that you added in step 1 so we can revert back to the default behavior of resuming the index build.</li>
</ol>
For any further questions, reach out to us at support@scalegrid.io.

Happy indexing!