Improve search time in MongoDB using Indexing and full-text search - Part 1
Tags: mongodb, node
Published: 2020-12-28 17:06:29

Consider a scenario, you launched a social media app and it's doing great. Within a week or so you managed to get around 30k users worldwide. Excited right! but things started going wrong. The feature where one user can search other users is taking more time to respond than expected, which may lead to a bad UX. And you know what users hate to wait.<br>
![waiting](https://media.giphy.com/media/SWVzkIlHdEckF81gnA/giphy.gif) 
That's where **indexing** can save your life. Indexing is a technique used in databases which help to optimize search or disk reads in a database while processing a query.
Enough of theory to understand now let's jump into code.
We will work with data of 10k pets. Each document has fields name, type, breed, age and index. Download the sample dataset from [here.](https://github.com/bera5186/blogs/raw/master/1.Improving%20search%20in%20mongoDB%20using%20indexing%20and%20full-text%20search/data.json) <br>
To import this json file into a collection fire the command in your terminal.
```code
mongoimport --db test --collection pets --authenticationDatabase admin --username <user> --password <password> --drop --file ~\downloads\data.json

```
This imports data.json file into test database with pets as collection name.
Now try out a search query, let's find pets with name Fido. You can run these commands in terminal or use a GUI tool like MongoDB compass or robo3T.
```code
db.pets.count({ name: 'Fido' })
```
This query returns the number of pets having a name Fido. Now let's see how much time is taken to execute and fetch results for this query. This can be done using mongoDB's [cursor.explain](https://docs.mongodb.com/manual/reference/method/cursor.explain/).
```code
db.pets.find({ name: 'Fido' }).explain("executionStats")
```
![output of above query](https://dev-to-uploads.s3.amazonaws.com/i/fq0oltyx0ws0rb7veorf.png)
In this screenshot, you can see executionTimeMillis: 13 which shows this query takes 13ms. In winning plan, it shows stage: 'COLLSCAN' which means it would sequentially scan all 10k documents in the database. Imagine running such a query with millions of documents in the database. Mind blown right! it would take forever to respond to such query. <br>
![mind blown](https://media.giphy.com/media/5aLrlDiJPMPFS/giphy.gif)
Let's optimize this by creating an index. An index is essentially a search optimized data structure ( B/B+ Tree ).
```Code
db.pets.createIndex({ name: 1 })
```
This command would create an index over the key name, however, you can make indexes over as many keys as you want. <br>
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/jp2w5p44p6qhp5tawqqo.png).

Now again fire the same search query and examine the execution stats.
```code
db.pets.find({ name: 'Fido' }).explain("executionStats")
```
The executionTimeMillis shows 3 which means this search now takes 3ms. From 13ms to 3ms, damn fast right! <br>
![fast](https://media.giphy.com/media/kBq78519uWUuG6k9g4/giphy.gif)
If you closely look at the results, you can observe a lot more changes. <br>
![After Indexing](https://dev-to-uploads.s3.amazonaws.com/i/w9pagjh5ga8qqhckcci5.png)
Here, in winning plan input stage, you can see stage:'IXSCAN' which means now this query scans keys of the index, not the entire collection. The result has a lot more information to observe. That's it for this post, in part 2 of this post I would show how you can implement full-text search using pure MongoDB without any library or something. 
Till then you can connect to me over [Linkedin](https://linkedin.com/in/rahul-bera-) or [Github.](https://github.com/bera5186).

*[cover](https://dribbble.com/shots/14746173-Early-Access-Illustration/attachments/6449060?mode=media) by [Gregory Taxerman on dribble](https://dribbble.com/GTX)*