A bot that tweets new DEV articles about Vue
Tags: vue, typescript, serverless
Published: 2019-03-03 10:41:33

---
title: A bot that tweets new DEV articles about Vue
published: true
description: Using an Azure Function to create a bot that scrapes dev.to for new VueJS articles and tweets them out!
tags: vue,typescript,serverless
cover_image: https://thepracticaldev.s3.amazonaws.com/i/9d87yce909m2qoxekbm4.png
---
So I love [VueJS](https://vuejs.org/). And I love dev.to articles about Vue. And I don't want to miss any. I could just keep constantly refreshing the [#vue tag](https://dev.to/t/vue) on dev.to, but instead I decided it would be more convenient to see all new Vue articles in my Twitter feed. So I wrote my a Twitter bot - [@TheVueDev](https://twitter.com/TheVueDev).

#Basic architecture and flow
The basic flow I decided on was the following
- An Azure Function written in Typescript runs every minute,
- The function scrapes [dev.to/t/vue/latest](https://dev.to/t/vue/latest) to check for new articles,
- If new articles are found, they get automatically tweeted!

An Azure Function is Microsoft Azure's "serverless" offering that allows you to run a small piece of code in response to some trigger. There's a generous free tier ([1 million free executions per month!](https://azure.microsoft.com/en-us/pricing/details/functions/)) that means this project is very cheap, though not entirely free as the Function requires an Azure Storage account to store some things related to the function. Still, the overall cost for this project is less than US$1/month.

#Setting up Twitter
Setting up the Twitter account is relatively straightforward (you just need to be prepared to answer a lot of questions that Twitter understandably asks to try to prevent abuse). After setting up a Twitter account for the bot at [twitter.com](http://twitter.com), I went to [developer.twitter.com](https://developer.twitter.com) and followed the prompts to apply for a developer account. Finally, I created a new "app", filled in some more details, requested some access tokens, and was presented with a page that looks like this:
![The Twitter developer page with some access tokens and secret keys to copy](https://thepracticaldev.s3.amazonaws.com/i/0chzhufxmibwequ1c9ej.png)
That's right, there are four tokens/keys, and we're going to need all of them!

#The Function
I followed the [quickstart guide](https://docs.microsoft.com/en-us/azure/azure-functions/functions-create-first-function-vs-code) in the Azure docs to create an Azure Function in Visual Studio code, selecting "JavaScript" as the language. I then ran `tsc --init` to add TypeScript, and was ready to start coding!

##Running code every minute
For each function you create, you need to specify what will trigger it (find all the options [here](https://docs.microsoft.com/en-us/azure/azure-functions/functions-overview#what-can-i-do-with-functions)). In my case I wanted to trigger the function every minute, which could be done with the following code in the function's `function.json` file:
```json
{
  "disabled": false,
  "bindings": [
    {
      "schedule": "0 * * * * *",
      "type": "timerTrigger",
      "direction": "in",
      "name": "everyMinuteTimer"
    }
  ]
}
```

##Scraping dev.to
I want to thank Sarthak Sharma for his amazing Vue CLI - check it out if you haven't already. The code he used to scrape dev.to gave me a lot of pointers for this project.
{% link https://dev.to/teamxenox/devto-cli-a-valentines-day-gift-to-my-favorite-people-56bd %}
Like Sarthak, I used the [x-ray library](https://github.com/matthewmueller/x-ray) to help with scraping. Every minute I want to load up [https://dev.to/t/vue/latest](dev.to/t/vue/latest) and scrape details of the latest VueJS articles. With the help of the Chrome devtools, grabbing things like the article title, author details, and a link to the article is quite straightforward - querying the page uses syntax very similar to CSS.
```typescript
const xray = Xray()
const articleScrapeData = await xray(
  'https://dev.to/t/vue/latest',
  '#substories .single-article',
  [
    {
      title: '.index-article-link .content h3',
      author: 'h4 a',
      link: '.index-article-link@href',
      tags: ['.tags .tag'], // need the tags so we can strip them out of the title
      authorLink: '.small-pic-link-wrapper@href'
    }
  ]
)
```
That will return an array of the articles on the page. There's a bit of cleanup of the data I needed to do ([here's the code](https://github.com/danielelkington/twitter-vue-dev/blob/master/AutoTweetDevArticles/article.ts) if you're interested). Of course, it makes sense to tweet out the author's Twitter handle if they've provided it, so I scraped that from the Author page.
```typescript
  const socialLinks: string[] = await xray(article.authorLink, [
      '.profile-details .social a@href'
  ])
  const twitterLink = socialLinks.find(x => x.includes('twitter.com/'))
  if (twitterLink != null) {
    // twitterLink is in the form https://twitter.com/{blah}
    const twitterURL = new URL(twitterLink)
    const twitterHandle = '@' + twitterURL.pathname.substring(1)
  }
```
For each article, I could then construct the tweet. I decided that tweets would contain the article title, then the author's twitter handle or name, and finally a link to the article (which dev.to very nicely formats with a beautiful card).
```typescript
const tweetContent =
`${article.title}

{ ${article.twitterHandle || article.authorName} }
${article.link}` // Yay for Template Literals!
```
Now all we need to do is Tweet the articles!
##The dilemma
The problem was that the function can't just tweet all the "recent articles" we find each minute, or we'll end up tweeting heaps of duplicates. Compounding the problem is that dev.to doesn't seem to provide an easy way to see when an article was published to anything more granular than the nearest day. I came up with three possible solutions:
1. Only run the function once every 24 hours, and tweet out any articles with yesterday's date. Problem is that if the function fails for some reason we'll miss a day, and if it works followers need to wait up to 24 hours for new articles!
2. Implement some sort of storage to keep track of the articles that have been tweeted. But this seems like a bit of overkill for a project like this. <sub><sup>(You could argue that Typescript is also overkill for a project like this, but I wanted to use it for learning purposes ☺️)</sup></sub>
3. Use the Twitter API to get the latest articles that the account has tweeted, do a comparison with the latest ones found on dev.to, and tweet out any that haven't yet been tweeted.

Option 3 seemed the most straightforward. Scraping [dev.to/t/vue/latest](https://dev.to/t/vue/latest) only returns the last 8 articles, while the Twitter API returns the last 200 tweets from the account, so the function should always be able to accurately determine whether an article has been tweeted.

##Using the Twitter API
Omar Sinan's article on building a Twitter bot was very helpful - check it out!
{% link https://dev.to/omarhashimoto/how-to-build-a-simple-twitter-bot-in-17-lines-ofcode-2aan %}
Like Omar, I used the [Twit](https://github.com/ttezel/twit) library to help with tweeting. Connecting it to [@TheVueDev](https://twitter.com/TheVueDev) was as simple as pasting in the four keys and tokens from the Twitter app page (as environment variables of course!).
```typescript
this.t = new Twit({
  consumer_key: process.env.twitter_consumer_key,
  consumer_secret: process.env.twitter_consumer_secret,
  access_token: process.env.twitter_access_token,
  access_token_secret: process.env.twitter_access_token_secret
})
```
Then we just tweet out our articles if they haven't yet been tweeted!
```typescript
// Get recent tweets
const res = await this.t.get('statuses/user_timeline', {
  screen_name: 'TheVueDev'
})
const recentTweets = <any[]>res.data
// Determine if article has been tweeted yet
// Need to look at entities.urls.expanded_url because Twitter 
// automatically shortens the URL in the tweet itself.
if (recentTweets.find(x =>
  x.entities.urls.find((u: any) => u.expanded_url === articleLink)
  ) == null) {
  // Article hasn't yet been tweeted - tweet it!
  await this.t.post('statuses/update', { status: tweetContent })
}
```
(Find the more efficient version of this code [here](https://github.com/danielelkington/twitter-vue-dev/blob/master/AutoTweetDevArticles/tweet.ts).)

#Deployment
Deploying the function to Azure can be done right from within Visual Studio code with a few button clicks using the [Azure Functions extension](https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-azurefunctions). Neat!

And with that, I have a functioning Twitter bot, and can now easily know about all the new VueJS articles as they appear on dev.to!

![A tweet from my Twitter bot](https://thepracticaldev.s3.amazonaws.com/i/y0qnn82nsfn8z2rvd65t.png)

[Here's the Twitter account](https://twitter.com/TheVueDev), and [here's the full code](https://github.com/danielelkington/twitter-vue-dev), with step-by-step instructions for deploying your own bot to tweet out any dev.to tag of your choice!
{% github danielelkington/twitter-vue-dev %}