Scammers signed up, scammed us of $870 in 15 minutes.
Tags: hack, security, twilio, scammers
Published: 2020-05-26 13:16:59

**No data or access was compromised.**

__On the 20th of May at 6:45 AM__, I got a phone call alert from [Spike.sh](https://spike.sh) from our [Twilio](https://twilio.com) integration about a surge in calls. 

Scammers abused our phone verification by otp generating thousands of phone calls. This qualifies as [Toll fraud](https://www.twilio.com/learn/voice-and-video/toll-fraud)

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/jz1pn5dgzl1xzao3q74y.png)

------------------
##Timeline of how it went down
* [7 AM](#7-am)
* [7:15 AM](#715-am)
* [7:30 AM](#730-am)
* [8 AM](#8-am)
* [9 AM](#9-am)
* [10 AM](#10-am)
* [11 AM](#11-am)
* [1 PM](#1-pm)
* [3 PM](#3-pm)
* [What did we learn?](#learn)
* [Resources](#resources)

------------------

##<a name="7-am">7 AM 👨‍💻</a>
Within the next 15 minutes, I was awake and glaring into my screen trying to find out what the hell is happening. 

I learn about these keys facts - 

* 40 new registrations in the last one hour using disposable email addresses, mainly [Mailinator](https://mailinator.com) (.com and .net)

* Our Twilio account had been recharged _which was unusual_

* So far, I was knee-deep with $107 bill on Twilio because scammers had made about hundreds of phone calls to a number in [Moldova](https://en.wikipedia.org/wiki/Moldova)

* Also, Twilio, being awesome, detected suspicious activity and blocked several phone calls by that time. I had emails from their Fraud detection team. __Thank you Twilio__

------------------

##<a name="715-am">7:15 AM 😧</a>
After digging into logs, I learn that the scammers were generating phone calls from our Phone verification section. We make phone calls to verify your phone number after signing up on Spike. I knew this was not the best idea going in too... 🙁 We now send SMS to verify instead of making a phone call.

There were two such sections. Both of them now look like this -

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/3c3rld0cfqn69c3nvixq.png)

------------------

##<a name="730-am">7:30 AM 💸</a>
I got a call from our credit card company who inform us we have been charged about __$870 in total so far__ 😭

We have barred making calls to Moldova and some other regions scammers had been targeting using [Twilio's GeoPermissions](https://www.twilio.com/docs/sip-trunking/voice-dialing-geographic-permissions)

By default, Twilio blocks voice call to some high-risk areas. 

![Twilio blocking high risk areas](https://dev-to-uploads.s3.amazonaws.com/i/nlbq926lti64y6jvi1xv.png)

And this is us blocking __Moldova__. 

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/l3yolbcyostfz5blmtdl.png)

All in all, about 2400+ phone calls were made so far.

------------------

##<a name="745-am">7:45 AM 🤖</a>
Integrate Google reCaptcha on the dashboard. Block spam user's accounts. I was personally very sceptical if things would go back to normal.

_When you do this, make sure to authorise the request on the backend too._

------------------

##<a name="8-am">8:00 AM 🚫</a>
So far, I have made about 20 deployments to production, thanks to GitHub Actions for automating that.

Replace phone call verification with SMS verification. This took some time along with testing.

------------------

##<a name="9-am">9:00 AM 📖</a>
Revoke all Twilio tokens and read their guide again on [toll frauds](https://www.twilio.com/learn/voice-and-video/toll-fraud). Are we missing anything? 

_Facepalm_ - Twilio has explicitly written in their docs to not use a phone call for verification via OTP and that's exactly what we did 🤦‍♀️

_We released some of our old phone numbers and bought new ones. This didn't take long._ 

------------------

##<a name="10-am">10:00 AM 🧟‍♂️</a>
Deploy to production but we still see a number of signups coming in via bots using Mailinator email addresses.

------------------

##<a name="11-am">11:00 AM 🕵️‍♀️</a>
Our search for a service API to tell disposable email addresses apart has begun. 

Shortlisted:

1. [https://block-temporary-email.com](https://block-temporary-email.com/)
2. [Kickbox.com](https://docs.kickbox.com/docs)
3. [NameAPI](https://www.nameapi.org/en/live-demos/disposable-email-address-detector)

{% stackoverflow 10976706 %}

__The problem is creating newer disposable email addresses is easy. It's perhaps too much to expect that one service would cover all of them.__

Interesting stat - A ton of our users were using Google OAuth, so we made the bold move. 

###Allow only Google OAuth. Remove login and signup using Email/Password combinations.

------------------

##<a name="1-pm">1:00 PM 👮‍♂️</a>
Time to take extra measures - 

* Activate Cloudflare [DDoS protection](https://www.cloudflare.com/ddos) with __I'm under attack mode__.

Cloudflare is pretty awesome at this. When you visit [Spike.sh](https://spike.sh), Cloudflare will try to determine if you are human or a bot. Sometimes, a test also appears for verification using hcaptcha. 

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/gkro65x5vo9oa1lntxa8.png)

* We installed [Needle.sh](https://needle.sh), which provides a security layer with their NPM module. In their dashboard, we noticed that security scanners have been probing our web properties for vulnerabilities multiple times a day. This made us realise the importance of using security tools for our web products from an early stage, __obviously__.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/lcce3hmqnbilmdhrgx2o.png)

* [Rate limiting](https://nordicapis.com/everything-you-need-to-know-about-api-rate-limiting) was super helpful and easy to setup. I have seen tons of __429 response status__ after we activated this.

* Setup [usage triggers](https://www.twilio.com/docs/usage/api/usage-trigger) on twilio

##<a name="3-pm">3:00 PM 🤞</a>
All of our measures are now live. Now - it's all about monitoring carefully. 

Scammers were unable to create new accounts. Yipee!
Some literally created accounts using Google OAuth and tried to make more calls but over 99% of them ended up in either 401 or 429 response codes. 

##<a name="learn">What did we learn?</a>
* Setup critical alerts, had I not gotten the phone call from [Spike.sh](https://spike.sh), I easily would have gotten a bill of more than $5000 by the time I woke up.
* Security is super important. Never too early to start.
* Use the right tools for the right job, we are using Cloudflare and Needle.sh
* Better safe than sorry

##<a name="resources">Resources</a>
These are some good reads I came across during the entire ordeal. Hope this helps you -

* [Cloudflare DDoS](https://cloudflare.com/ddos)
* [Needle.sh](https://needle.sh)
* [SSL error with Letsencrypt](https://community.cloudflare.com/t/err-ssl-version-or-cipher-mismatch-appears-with-cloudflare-letsencrypt-and-wordpress/79097)
* [https://block-temporary-email.com](https://block-temporary-email.com/)
* [SO question](https://stackoverflow.com/questions/10976706/how-to-block-disposable-email-addresses-in-your-websites-registration-form)
* [Detecting headless browsers](https://antoinevastel.com/bot%20detection/2018/01/17/detect-chrome-headless-v2.html)
* [Impossible to detect headless browsers](https://www.google.com/search?q=detecting+headless+browser+is+impossible&oq=detecting+headless+browser+is+impossible&aqs=chrome..69i57.6513j0j7&sourceid=chrome&ie=UTF-8)

_This doesn't cover all the steps we have taken to stop abuse or DDoS._