I Accidentally wiped the entire dataset in the Production database. Here's what happened next
Tags: webdev, career, javascript, showdev
Published: 2020-11-17 18:33:02

One of the tragic accident in my job turned out to be good learning for me in recent days. Yes, I accidentally wiped the entire dataset in the production database.

Before, getting into the details of it. let me tell you a bit about myself, I am a Full Stack Engineer specialized in React, Nodejs. I am working in a startup where we use Nodejs, Postgres in production.

Recently, i was working in an API which updates the database table. table has structure like this

```jsx
module.exports = (sequelize, Sequelize) => {
    const User = sequelize.define("user", {
      name: {
        type: Sequelize.STRING
      },
      email: {
        type: Sequelize.STRING
      },
      data : Sequelize.JSON,
    });
  
    return User;
  };
```

an important thing to note here is the `data` that is in the JSON structure. API that I built should update the data object in the table.

To give you an example, Here's the structure of data on GET request

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/3rvue2t0o7cvgmeiplmc.png)

it contains the `data` in JSON structure which has fields such as `phone`, `address`, `country`, and `postal code`.

So, I need to write an API to change the phone number inside of the data. I thought it's kind of easy one to do and did that in a few steps.

```jsx
app.post('/phone-number/update/:id',async (req,res) => {
    try {
        const id = req.params.id;
        const phonenumber = req.body.phonenumber

        const user = await User.update({ "data.phone" : phonenumber  },{
            where : {
                id
            }
        })

        res.status(200).json(user)
    }
    catch(e){
        res.status(500).json(null)
    }   
})
```

it updates the phone number inside of the data object. everything looks good so far. After that, I wrote a test-case to cover the scenario where the updated phone number is stored in the DB and it passed.

Everything looks good so far. So, it went to production. i didn't realize the disaster that is going to happen on production. i was in my home like,

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/j3s30q7kvlem2mtphaoz.gif)


Here's what happened in production, the API entirely wiped the JSON data replacing them with just `phone number`. i was shocked to see this,

To give you an example, Here's the data before and after the API

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/t2zvmkjcucoe36sors2e.png)

After the API update,

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/7t83l2ydvuclfezl8mah.png)

Then, i realized that postgres update on `JSON` shouldn't work like this. Luckily, our database took a backup exactly before this came up. so, i just rolled back the database data and commented out the functionality in this API.

but, my CTO noted this issue and helped me to resolve this one. In the meantime, I found an another way to fix the API functionality. it will be something like,

 

```jsx
app.post('/phone-number/update/:id',async (req,res) => {
    try {
        const id = req.params.id;
        const phonenumber = req.body.phonenumber

        let user = await User.findOne({ id })

        user.data = { ...user.data,phone : phonenumber }
        await user.save()

        res.status(200).json(user)
    }
    catch(e){
        res.status(500).json(null)
    }   
})
```

I know it doesn't look like an effective solution. but at that moment, I wanted a working solution that should be deployed. I quickly wrote test-cases to cover the scenario and raised a PR. CTO personally looked into this code and reviewed it and gave some feedback. finally, the patch was merged into production that day.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/qansggj64wkzvtcnbmvx.gif)


it may look like a small & silly issue. but, it has the potential to wipe out the entire dataset. Instead of worrying about this incident. I took it as good learning for me. they are,

- Even if we write test cases to cover the situation. it's always one step better if we can manually test the situation.
- Never get stressed if your code broke the production server. I know it shouldn't happen. In some worst cases, it will. So, take it as a learning curve and avoid that mistake in the future.
- Things like this will happen, At that moment, Having a supporting colleague or seniors will help a lot. That comes along in our careers.

I hope you learned something from this real-world experience. I wanted to share my experience so that you can get that experience without you need to undergo the same situation. we will see in another real-time experience. until then, Happy Coding :-)