Building a netlify lambda function to turn twitter lists into email digests.
Tags: twitter, javascript, netlify, serverless
Published: 2020-04-07 15:06:19

<!-- wp:paragraph -->
<p>I'm probably not the only one who feels overwhelmed by the overall panic on social media right now.</p>
<!-- /wp:paragraph -->

<!-- wp:quote -->
<blockquote class="wp-block-quote"><p>I've decided to code some sanity back into my life by creating a netlify lambda function that emails me tweets once a day.</p></blockquote>
<!-- /wp:quote -->

<!-- wp:paragraph -->
<p><a href="https://github.com/tobenxe/twitter-digest">Github Link</a></p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>Tutorial: Setting Up</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Things we need</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li>A free sendgrid account and API key</li><li>A netlify account</li><li>A twitter API key and API secret</li><li>A public twitter list</li><li>A cron job service that allows you to set headers (what I found: <a href="http://cron-job.org">cron-job.org</a> — free, <a href="http://easycron.com">easycron.com</a> - free trial) or if you have a server you should be able to set this up yourself.</li><li>Some understanding of Javascript &amp; HTTP.</li><li>An email</li></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>We need to create a node project and install dependencies. Run the following commands in your project directory. You should have a package.json/package-lock.json file and a node_modules folder afterwards.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>npm init -y
npm i netlify-lambda @sendgrid/mail encoding twitter-lite dotenv-webpack
</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>netlify-lambda - helps with  bundling and local development.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>@sendgrid/mail -  helps us use  sendgrid's mail API</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>twitter-lite - a wrapper for twitter's API</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>dotenv-webpack -  setting local development environment variables (not covered in this post check out the github repo)</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>encoding -  was installed to solve a deployment error I encountered (<a href="https://github.com/netlify/netlify-lambda/issues/142#issuecomment-483880089">issue</a>).</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>Project folder structure</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Create the following files according to the structure below. (node modules, package-lock.json and package.json should already be there)</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>node_modules/ 
src/
  lambda/
    digest.js 
  utils/
    twitter.js 
    sendgrid.js 
    email.js 
netlify.toml 
.gitignore 
package.json 
package-lock.json 
</code></pre>
<!-- /wp:code -->

<!-- wp:heading -->
<h2>Creating the digest</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>We have to do 3 things (Only 2 of which require writing code):</p>
<!-- /wp:paragraph -->

<!-- wp:list {"ordered":true} -->
<ol><li>Get tweets from a twitter list.</li><li>Send out an email containing those tweets.</li><li>Schedule emails to be sent every day (or whenever) with <a href="http://cron-job.org">cron-job.org</a> </li></ol>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>Below is our lambda function (digest.js) which brings together the different components of the function.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>The first thing we do is add some basic authorization and validation of the request. A valid request to the function needs to:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li>Have a request method of POST</li><li>Have an authorization header</li><li>Have the proper value in its authorization header</li></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>NOTE: I will be using process.env to refer to variables that would be set later on when deployed.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>I am returning the status code of 404 in order to avoid revealing whether the endpoint exists. My rationale is that 404 - Not Found is more likely to deter some bored spammer/attacker than 401 - Unauthorized. </p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>//digest.js
const twitter = require('../utils/twitter');
const sendGrid = require('../utils/sendgrid');

exports.handler = async (event, context)=>{
         try{
             //Validate the request method and the authorization header
             if(event.httpMethod != 'POST') return {statusCode: 404}
             if(!event.headers.authorization)  return {statusCode: 404};

             //check for valid authorization value
             const basicAuth = (new Buffer(`${process.env.AUTH_USER}:${process.env.AUTH_PASS}`)).toString('base64')
             if(event.headers.authorization.split(' ')&#91;1] !== basicAuth) return {statusCode: 404};

             //Get tweets
             const tweets = await twitter();

             //Send email if there are tweets available
             if(tweets.length > 0) await sendGrid(tweets);

             // success
             return {statusCode: 200}
             
         }catch(err){
             console.log(err)
             //error
             return {statusCode: 500}
         }
}
</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>The rest of the function involves calling the modules we imported at the top. One is a function that authenticates with twitter and gets tweets from our list (twitter.js). The other sends out our email (sendgrid.js).</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>1) Getting tweets from our twitter list</h3>
<!-- /wp:heading -->

<!-- wp:code -->
<pre class="wp-block-code"><code>//File - twitter.js
const Twitter = require('twitter-lite')
module.exports = async () => { 
    //a) authenticate with twitter
    const user = new Twitter({
      consumer_key: process.env.TWITTER_KEY,
      consumer_secret: process.env.TWITTER_SECRET
    });
    const response = await user.getBearerToken();
    const app = new Twitter({
      bearer_token: response.access_token
    });

     //b) get tweets from list
    const data = await app.get('lists/statuses', {
                                list_id: process.env.TWITTER_LIST_ID,
                                tweet_mode: 'extended'
                            })

     //c) get only the necessary data &amp; return tweets
    const tweets = data.map(tweet=>{
        return {
            text: tweet.full_text, 
            user:tweet.user.screen_name,
            url: `https://twitter.com/${tweet.user.screen_name}/status/${tweet.id_str}`
        }
    })

    return tweets;
     
}
</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>a) We installed a package called twitter-lite which is a wrapper for the twitter api. To authenticate we just need our twitter api key (TWITTER_KEY) and twitter api secret (TWITTER_SECRET) passed into the packages Twitter function. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Twitter's API has user authentication and app authentication. We are using app authentication we just need a bearer token to start making requests to twitter's api. The package takes care of basically everything, you can see the full documentation for twitter-lite <a href="https://github.com/draftbit/twitter-lite">here</a></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>b) We can use twitter-lite to get tweets from a list by calling the get method on "app". The object being passed into app.get( ) contains the query parameters needed for <a href="https://developer.twitter.com/en/docs/accounts-and-users/create-manage-lists/api-reference/get-lists-statuses">twitter's lists/statuses endpoint</a>. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>This includes the id of the twitter list(list_id) and whether we'd like to get the full tweet versus a truncated version of the tweet (tweet_mode). It returns an array of tweet objects when done.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>c) Once we have the tweets, we only need a few bits of information (the tweet text, the username of the "tweeter" and the tweet URL). Using the array map method we can create a new array that only contains the necessary data from each tweet.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>2) Sending the email.</h3>
<!-- /wp:heading -->

<!-- wp:code -->
<pre class="wp-block-code"><code>//sendgrid.js
//a) set sendgrid's api key
const sgMail = require('@sendgrid/mail');
const createHtmlEmail = require('./email');
sgMail.setApiKey(process.env.SENDGRID_KEY); 

module.exports = (tweets) => {
    return sgMail.send({ 
            to: process.env.MY_EMAIL,
            from: process.env.MY_EMAIL,
            subject: 'Your Tweet Digest',
            html: createHtmlEmail(tweets), //b) create html email &amp; send
            });
}
</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>a) Here we import <a href="https://github.com/sendgrid/sendgrid-nodejs/tree/master/packages/mail">sendgrid's npm package</a>, set the api key for sendgrid and import the module that helps us create the HTML email containing the tweets.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>b) sgMail.send( ) takes in an object with some information about the email we need to send. To get the value for the "html" property we call createHtmlEmail and pass in our tweets.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Here is what goes on inside the createHtmlEmail function. We are essentially inserting our tweet data into html and returning a string of html. This is very easy to do thanks to <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Template_literals">template literals</a>.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>//File - email.js
//creates the html email
module.exports = (tweets) => {
    let tweetElements = '';

    //insert tweet data into elements
    tweets.forEach(tweet=>{
       tweetElements += `
                &lt;div style = "width: 100%; border: 1px solid black; margin-bottom:10px; padding: 10px ">
                &lt;p>@${tweet.user}&lt;/p>
                &lt;p>${tweet.text}&lt;/p>
                &lt;a style="color: white; padding:5px; background: black; text-decoration:none" href="${tweet.url}">Read Tweet &lt;/a>
                &lt;/div>
               `
    })
    //Wrap tweet elements in a div
    let email = `   
        &lt;div style = "width: 40rem; font-size: 1.2rem; margin: 0 auto">
            ${tweetElements}
        &lt;/div>
    
    `;
 
    return email;
}
</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>sgMail.send() returns a promise which is why we use the await keyword in our lamda function(digest.js).</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>Deploy the function !</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Before we deploy, we  need to edit a few files</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li>package.json - adding a build script </li><li>netlify.toml - adds a build command and lets netlify know what folder our function is in (src/lambda)</li><li>.gitignore - removes our node_modules folder from git tracking</li></ul>
<!-- /wp:list -->

<!-- wp:code -->
<pre class="wp-block-code"><code>//package.json
"scripts": {
    "build:lambda": "netlify-lambda build src/lambda"
}

//netlify.toml
&#91;build]
  Command = "npm run build:lambda"
  Functions = "lambda" 

//.gitignore
node_modules/</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Now we can commit the project to github.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>git init
git remote add origin &lt;link to git repository>
git add .
git commit -m "commit message"
git push origin master</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>We can now login to netlify. You should see the option to create a new site from git, once you click that you'll go through a 3 step process where you choose your git provider, authorize netlify and pick a repository. The most important is step 3 where we have build options.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Clicking "Show Advanced" then "new variable" should let you add environment variables (the process.env prefixed stuff throughout the code).</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Here is a list of the variables that need to be set</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>Name:           Description:
TWITTER_KEY     your twitter API key
TWITTER_SECRET  your twitter API secret
AUTH_USER       a username (does not have to be your twitter username, just come up with something)
AUTH_PASS       a password (does not have to be your twitter password, just come up with something)
YOUR_EMAIL      your email
TWTTER_LIST_ID  id of your twitter list, it is simply the numbers at the end of the URL when you are on your twitter list
Example:  &lt;https://twitter.com/i/lists/1234> will have a TWITTER_LIST_ID of 1234
</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>The <a href="https://github.com/tobenxe/twitter-digest#usage">github repository</a> explains some of these environment variables a bit better. After filling in the environment variables the project can be deployed.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p></p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>Scheduling the function to run every day.</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>You should be able to access your function's url like this:</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>https:&#47;&#47;&#91;DEPLOY_URL]/.netlify/functions/digest.js
</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>[DEPLOY_URL] represents the site url shown in netlify's overview dashboard for your deployed site. For example if your site's url is <a href="http://example.netlify.app">example.netlify.</a>com then your function url is:</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><a href="https://example.netlify.com/.netlify/functions/digest.js">https://example.netlify.com/.netlify/functions/digest.js</a></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>We'll be using a cron job on <a href="https://cron-job.org">https://cron-job.org</a> to call the function's url daily. After signing up on the site you should be brought into a members panel where you can click "Cronjobs" on the upper left navigation bar then click "create cronjob".</p>
<!-- /wp:paragraph -->

<!-- wp:image {"sizeSlug":"large"} -->
<figure class="wp-block-image size-large"><img src="https://i.imgur.com/Ai7hmMa.png" alt=""/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Now fill out the form</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li>Title - Any name you want</li><li>Address - your function url (https://....)</li><li>Check "Requires HTTP authentication"<ul><li>User - same as AUTH_USER set previously</li><li>Password - same as AUTH_PASS set previously</li></ul></li><li>Schedule - set the schedule for any time interval (I'm using every day)</li></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>When you are done with that, click "create cronjob". You'll be redirected to a page listing your current cronjobs, now find the newly created job and click "edit".  </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>In the edit form page you can scroll down to find "Advanced Settings" which allows you to change the request method from GET to POST then save the form.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>and.....</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p> That's it! </p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>Results! :D</h2>
<!-- /wp:heading -->

<!-- wp:image {"sizeSlug":"large"} -->
<figure class="wp-block-image size-large"><img src="https://i.imgur.com/Px4vvMY.gif" alt=""/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Looks decent.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>There are lots of improvements to be made of course, so if you have some ideas to improve it you can go ahead and contribute to the project on <a href="https://github.com/tobenxe/twitter-digest">github</a>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>: D</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p></p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>Resources:</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p><a href="https://github.com/tobenxe/twitter-digest">Github Repository</a></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><a href="https://tobenxe.com/building-a-netlify-lambda-function-to-turn-twitter-lists-into-email-digests/">Original Blog Post</a></p>
<!-- /wp:paragraph -->