Solving life problems with JavaScript
Tags: javascript, devtools, node, devops
Published: 2019-10-22 23:52:47

A story of how, how I’m a big dumb dummy, but JavaScript is good.

<small>(originally posted on [Medium](https://medium.com/@phenomnominal/solving-life-problems-with-javascript-7c61eaf2be13))</small>

## TL;DR: 💯

I didn’t make an important appointment, and there were none available for next month… So I used JavaScript to find one within a few days! I used [Chrome DevTools](https://developers.google.com/web/tools/chrome-devtools/) to figure out how the appointment booking website work, and wrote some JavaScript to email me when appointments became available!

## Surprise, surprise. I messed up. 🤪
I recently moved from New Zealand 🇳🇿 to Sweden 🇸🇪, and to be honest with you, it’s been pretty straightforward! I was lucky enough to get a role where a relocation agency organised almost my entire relocation. I had to fill out a few forms and pack my stuff. In fact, so much of it was handled for me that I didn’t do my one super important task— book a biometrics appointment at ***Migrationsverket*** in Stockholm. This appointment is a critical step in the migration process that would allow me to get a ***personnummer***. That number would allow me to open a Swedish bank account, and start getting paid! People book these appointments weeks and weeks in advance, so it wasn’t like I could walk up and get it sorted.

To make this a bit more embarrassing for me, my girlfriend had been trying to find a time for her own appointment. When she asked me if I needed to do the same, I assured her (wrongly 😅) that the relocation agency would sort out mine. She had been having trouble finding an appointment on the website. We thought the website didn’t work, but in reality, the appointments are very scarce. They get snapped up as soon as they become available!

![As I’m writing this (June 21st, 2019), the next available appointment in Stockholm is July 17th!](https://thepracticaldev.s3.amazonaws.com/i/w1830dnr6r06px8uribm.png)

I wanted to get it sorted out as soon as possible (cause money 💰)! I checked if I could get an appointment in another city, or if I could travel to another country. My relocation advisor reassured me that people often cancel their appointments, so I should keep checking. There are 14 different appointment locations in Sweden, and it takes six clicks to get the list of available times! That means it takes a bit of time to go through and look at all the options. I’m way too lazy for that, and scared that I would miss out cause I was looking at the wrong place at the wrong time.

## I can automate that! 🤖

Luckily for me, I write code in the same way that I live my life — badly. I tend to persevere when it comes to figuring out how things work. I’ve had to build up a set of tools for getting myself out of situations like this one. I was pretty confident that I’d be able to reverse engineer the website, and automate the process. I considered using [Puppeteer](https://github.com/GoogleChrome/puppeteer) to drive a browser and fill out the forms, but I thought there might be a simpler way. I decided to use HTTP requests, to get the data from the Migrationsverket website, and to send myself an email whenever a new appointment became available.

Figuring this out seemed like it would be quite straightforward, but turned out to be far from it. Let’s look through the steps I took to figure out what was going on.

### Step 1) Where does the data come from?

First, I used the **Network** tab, so that I could watch what requests the browser was making as I used it. I navigated to the website, filled out the form, and clicked the calendar to look at each week. The **Network** tab let me watch the requests as they happened. Each click triggered a request to `“BehaviorListener.1-form-kalendar"`:

![Image of the Network tab in Chrome DevTools, showing the requests for appointment data.](https://thepracticaldev.s3.amazonaws.com/i/ha97jppcsxt8gopj7gyl.png)

When I looked into each individual response, I saw the JSON data for the list of appointments coming back! Woohoo 🎉!

![Image of a specific response in the Network tab, showing the JSON appointment data.](https://thepracticaldev.s3.amazonaws.com/i/dnsbt06psnq0va7lpw9s.png)

I thought that I would be able to make a request to that URL and be done 😎… That looked something like this:

{% gist https://gist.github.com/phenomnomnominal/f2b0ff5867b3a33c90d19488ab841f1c %}

Alas, if only it was that easy. When I ran the above code, I got back a chunk of HTML, containing the following 😔:

```
“The requested page has expired. To countinue booking try to go back using the navigation buttons in your web browser.”

(Yes, “countinue” is in the real response)
```

### Step 2) How do I get the real data?

I knew where the data was coming from, but I couldn’t actually get it. So what was going on? I had a few clues. The response mentioned “expired”. This implied that there is some state involved. That hinted to me that there could be cookies involved, and I knew I could confirm this quite easily. By using the ***“copy as cURL”*** tool in DevTools, I could replay the request including all the cookies and see what I got:

![Image showing the “Copy as cURL” feature in DevTools. Copying this request means you can replay the whole request from the terminal.](https://thepracticaldev.s3.amazonaws.com/i/ojxxs6a6vax42vh2vuz8.png)

When I replayed the whole request including the cookies and other headers, I got back the expected response!

![Image showing the request replayed via cURL, and the JSON response of all the appointment data.](https://thepracticaldev.s3.amazonaws.com/i/frvs2np7vn9psqaw7ubs.png)

### Step 3) How do I get the data for different locations?

The next issue was that the original request URL didn’t contain an actual “query”. It has the `start` and `end` dates, but what about the location? I needed more information.

I went through the appointment process several more times, looking for clues. I noticed that no matter what location I selected, I always ended up with a URL in the following format:

{% gist https://gist.github.com/phenomnomnominal/bb49b5284002389e5725608576c78b76 %}

As I fiddled around a bit, I discovered that the last query parameter didn’t seem to matter . It was just the current timestamp. Of the remaining params, `start` and `end` are important, but they didn’t give any new information. The first parameter is interesting though — it changed with each new page request. Where was it coming from? The server must generate it, but how?

The next thing I tried was to look at the **Network** tab again, this time looking at the bigger picture. What requests does the site make throughout the whole process? I turned on the ***“Preserve log”*** and ***“Disable cache”*** options, so that I could see requests across the flow:

![Image of the Network tab in Chrome DevTools, showing all the requests made while completing the appointment process. Enabling the “Preserve log” option means that the list of requests isn’t cleared when you navigate between pages.](https://thepracticaldev.s3.amazonaws.com/i/0vx3ydfmh7dptxdq7rcm.png)

The short answer is that there are *lots* of requests. Unfortunately, I couldn’t find out anything useful here, so I had to try something else…

My next idea 💡 was to disable JavaScript. You can do that by going to the DevTools settings (the three vertical dots for ***“Customize and control DevTools”***, followed by ***“settings”***), and clicking the ***“Disable JavaScript”*** checkbox:

![Image of the “Disable JavaScript” checkbox in the DevTools settings.](https://thepracticaldev.s3.amazonaws.com/i/oxycqf3ur0bmkv3ty2kb.png)

Now when I tried to go through the process again, clicking the ***“Continue”*** button didn’t work! Google Translate was also broken 🤪! That verified that the form submit was being handled by JavaScript! Now I needed to explore and see exactly what it did. I used the ***“Inspect Element”*** tool to get a better look:

![Image showing the Element tab of DevTools, with the “Continue” button highlighted This reveals that the element has an id of “bokaSubmit”.](https://thepracticaldev.s3.amazonaws.com/i/g64nmccq0sy6pklw2bgg.png)

Inspecting the element revealed that it had an `id`, which meant I had another clue! I used the DevTools “Search” to look through the whole site for any references to that `id`:

![Image showing the Search panel in DevTools. There are 10 results for the “bokaSubmit” id.](https://thepracticaldev.s3.amazonaws.com/i/n39pbl6m2x8cbcbz9h3q.png)

The search revealed the there is a click handler attached to this button. That particular code looked like this:

{% gist https://gist.github.com/phenomnomnominal/8c5673dfd5a719bea7777b901b3b3331 %}

I added a breakpoint to see where the real redirect would go to:

![Image showing the Source tab of DevTools, with a breakpoint before the redirect happens.](https://thepracticaldev.s3.amazonaws.com/i/orj1mxom8r8d2bnk4zch.png)

And now I had more clues! A new URL, with a new page, and new query parameters. What happens if we make a request to that URL?

{% gist https://gist.github.com/phenomnomnominal/229f298d0be90a747745a0931b519467 %}

This time I got a real response 🥳:

```
Du har valt att boka tid för 2 personer för att lämna fingeravtryck och bli fotograferad hos NATIONELLT SERVICECENTER 1 STOCKHOLM
```

or

```
You have chosen to book time for 2 people to leave fingerprints and be photographed at NATIONAL SERVICE CENTER 1 STOCKHOLM
```

The response also contained the URL for `“BehaviorListener.1-form-kalendar"`. It even included the query parameter with the changing number values. It was looking pretty good!

I went through the appointment process with each of the different locations. This allowed me to deduce that the `enhet` query parameter controlled the location. I built up a little mapping for the different possible values:

{% gist https://gist.github.com/phenomnomnominal/ad67ca7eb6c6064cee565fcd41b48c30 %}

At this point I had three crucial bits of information:

  1. The format of the URL to request the generated calendar page. Requests to this URL also return the necessary session cookies.

  2. The format of the URL to request the JSON data for the appointments.

  3. The list of different locations for the enhet parameter.

### Step 4) How do I put this all together?

My little node.js script became a bit more complex. It creates a first request based off the location, then looks through the response to find the URL for `IBehaviorListener`. It then takes the random numbers from that, and then creates a second request based on the random number. The response is a set of appointments!

{% gist https://gist.github.com/phenomnomnominal/6f30a009d3fde3d31942581e36e891c5 %}

There’s a few little “tricks” here that make it work:

  1. By including `{ jar: true }`, it tells the `request` module to store cookies. That means that the second request doesn’t give us the “expired” message.
  
2. A tiny little Regular Expression to find the right data in the body of the first request. This is particularly brittle, and suspect to breaking if their code changes (😢).

### Step 5) How to get it running automatically?

I already had the basic functionality in place. All I needed to do was to make it loop over each of the locations and check if there were new appointments available. This involved some gross code (🤢), but hey, it worked!

{% gist https://gist.github.com/phenomnomnominal/4df87a3c2a6f377eca0e676b4c53a672 %}

Every five seconds, the script would make a request to the next location, and store the list of appointments in the `ALL_APPOINTMENTS` object. Unfortunately, the appointments endpoint sometimes returns nothing, even though are available appointments. The data eventually shows up, so we have another timeout that waits a few minutes before the script starts sending emails. It’s unfortunate that this is necessary, but this actually explains why my girlfriend was having such a rough time with the website! After two minutes, the information should be stable, and any “new” appointments should actually be new. This means we can send an email notification.

### Step 6) How do we send an email?

Thanks to the wonderful node.js ecosystem, this step was actually easy! I used [nodemailer](https://www.npmjs.com/package/nodemailer), and it worked wonderfully:

{% gist https://gist.github.com/phenomnomnominal/29447a79f0c2fd73130542370e2a202b %}

This sends emails to me from me, with a little alias (`+biometrics-appointment`) so I could filter them. The `SENDER_EMAIL` and `SENDER_PASSWORD` get set as environment variables. `SENDER_PASSWORD  had to be a generated [“app password”](https://support.google.com/accounts/answer/185833?hl=en) to get around my two factor.

This meant that each night, I’d receive quite a few emails 😅:

![Image showing what my email inbox would look like each morning, with dozens of emails for newly available appointments!](https://thepracticaldev.s3.amazonaws.com/i/gzsv2s5801lhma27qalz.jpeg)

## The whole thing:

And putting it all together, the whole script looked like this:

{% gist https://gist.github.com/phenomnomnominal/4263594f41ed77a488f835581239ad02 %}

This 123 lines of code meant that instead of waiting almost a month for an appointment, I had one booked in Stockholm within three days. I could also find an appointment for my girlfriend on a date that coincided with her visiting me. All in all, it was a great success!

## Wrapping up:

I made it to my appointment! I was a little bit late (of course 🙄), but it went fine, and now I have a biometrics card, and a bank account. JavaScript can’t solve most of my day to day mishaps, but it sure is nice when it can! DevTools is very useful, and the tricks we use to debug our own sites can also be used to figure out how other websites work 🚧.

Anyways, this was kind of dumb, but also kind of fun, and I hope you found it entertaining/useful. Please reach out to me on Twitter ([@phenomnominal](https://twitter.com/phenomnominal)) and let me know what you think!

🦄