Scrape Web App Data & Add it to Google Sheets with WayScript
Tags: wayscript, python, scrape
Published: 2020-06-17 13:22:49

Introduction
------------

Scraping data from Twitter is very useful when you want to make sentiment analysis or understand a phenomenon taking place online. However it requires a good level of coding and a better comprehension of changing websites structures. Using Wayscript, we can bypass these problems, and seamlessly integrates Twitter and Google Sheets. Today we’ll create a script to automatically scrape Twitter for social mentions add to Google Sheets.

Prerequisites
-------------

No prerequisites but some content you might find useful:

*   [Working Google Sheets](https://docs.wayscript.com/library/modules/google-sheets)
*   [Workig with Twitter](https://docs.wayscript.com/library/modules/twitter)

Building our script
-------------------

Our workflow will look like this:

*   Add a time trigger to periodically run the Twitter Scraper module
*   Use the list returned by the module and loop through it to get each item
*   Use important data about each item and add it to sheet files

You can work with any trigger you want like a text message trigger or Gmail trigger, but here we’ll use a time Trigger.

![Adding Time Trigger](https://cdn.wayscript.com/blog_img/106/1.png)

The Twitter Module
------------------

The next step is to scrap Twitter according to a special mention or search item. For this purpose, we’ll use Twitter Module provided by Wayscript to get the latest 50 tweets about cryptocurrency.

![Add Twitter Trigger](https://cdn.wayscript.com/blog_img/106/2.png)

Set the mode to Search for word or #hashtag. In advanced input parts, you can select options such as ignoring retweets, replies and choose the result type (mixed or popular).  

![Setting input search](https://cdn.wayscript.com/blog_img/106/3.png)

We've selected the Tweet outputs to get a list. It will be more easier to work with. You'll notice some changes in variables panel.

![Variables panel](https://cdn.wayscript.com/blog_img/106/8.png)

Add to Google Sheet
-------------------

The Twitter module will return a list of 50 tweets, each item of the list containing a lot of information. Here, we will only use the username, the name, the content, and the date.  
But first, we must add a loop step.

![Adding loop step](https://cdn.wayscript.com/blog_img/106/4.png)

Then in the loop add the Google Sheet module and set it to Write Mode.  

![Google Sheet configuration](https://cdn.wayscript.com/blog_img/106/5.png)

Now let’s add the variables. Understand that here each variable is a column.

![Adding variables](https://cdn.wayscript.com/blog_img/106/6.png)

Final Step
----------

Now turn on the time trigger. The automation of this script is handled by Wayscript. Then, the google sheet will be updated every day at 11.00 am.

![Turn on Time Trigger](https://cdn.wayscript.com/blog_img/106/7.png)

Questions or concerns?
----------------------

Questions about this script or anything else? Join our [discord](https://discord.gg/VWbXSbj). We're always around to help. If you want to work the full script template, just find it [here](https://wayscript.com/user/mangabo_kolawole/itGYUEsi).