Introduce to my Taiwan COVID-19 news website and crawler
Tags: actionshackathon21, github, opensource
Published: 2021-12-07 15:27:28

---
title: Introduce to my Taiwan COVID-19 news website and crawler
published: true
description: My submission to the GitHub Actions x DEV Hackathon 2021!
tags: actionshackathon21, github, opensource
---

## Instructions

In Taiwan, it's COVID-19 outbreak on May 13 and there're about 100 local cases every day. And it's zero local case since October.

At that moment, I'm really concerned about daily local cases so I develop this project and publish it to the [GitHub](https://github.com/peter279k/covid19-tw-news). I use the cron job feature to fetch the latest COVID-19 news and publish my new web page during GitHub action running. 

### My Workflow

My completed workflow files are available [here](https://github.com/peter279k/covid19-tw-news/tree/master/.github/workflows) and it includes following YAML files:

- `fetcher.yml`, this YAML file is for fetching latest COVID-19 news and store them to be JSON file. This web crawler is written in Python. And It will be run every hour.
- `web_build.yml`, this YAML file is for publishing new website and loading new JSON file. This website is written in JavaScript with Vue.js framework. And it will be run every seventy minutes.

### Submission Category: 

This category is for DIY-Deployments and I hope developers love this and can be referenced :-).

### YAML files and link to repository
Here are repository preview and completed codes about above two YAML files:

{% github peter279k/covid19-tw-news %}

- `fetcher.yml`

```YAML
name: Update CSV Dataset Records from Taiwan CDC website

on:
  schedule:
    - cron: '0 * * * *'
  workflow_dispatch:

jobs:
  build-and-deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Get working copy
        uses: actions/checkout@master
        with:
          fetch-depth: 1
      - name: Set up Python 3.8
        uses: actions/setup-python@v2
        with:
          python-version: '3.8'
      - name: Install Requirements
        run: pip install -r requirements.txt
      - name: Update COVD-19 Labs
        run: python Fetcher/LabFetcher.py
      - name: Update MOHW RSS News
        run: python Fetcher/MohwRssFetcher.py
      - name: Update COVID-19 global cases and deaths
        run: python Fetcher/GlobalCasesFetcher.py
      - name: Update COVID-19 TW datasets
        run: python Fetcher/DailyCovid19TW.py
      - name: Copy JSON files to assets folder
        run: cp datasets/*.json web/src/assets/
      - name: Commit and push if it changed
        run: |
          git config user.name "peter279k"
          git config user.email "peter279k@gmail.com"
          git add -A
          timestamp=$(date -u)
          git commit -m "Last Commit: ${timestamp}(TW)" || exit 0
          git push origin master
```

- `web_build.yml`

```YAML
name: 'Web Build and Deploy'

on:
  schedule:
    - cron: '10 * * * *'
  workflow_dispatch:
  push:
    branches:
      - main

jobs:
  build-and-deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout 🛎️
        uses: actions/checkout@v2.3.1
        with:
          persist-credentials: false

      - name: Install and Build 🔧
        working-directory: web
        run: |
          cp .env.example .env
          export VUE_APP_MAPKEY=${{ secrets.VUE_APP_MAPKEY }}
          sed -i -e "s/MAP_KEY/$VUE_APP_MAPKEY/g" ./.env
          npm install
          npm run build
          cd dist
          cp index.html 404.html

      - name: Deploy 🚀
        uses: JamesIves/github-pages-deploy-action@3.6.2
        with:
          GIT_CONFIG_NAME: "peter279k"
          GIT_CONFIG_EMAIL: "peter279k@gmail.com"
          SINGLE_COMMIT: true
          GITHUB_TOKEN: ${{ secrets.TOKEN }}
          BRANCH: gh-pages
          FOLDER: web//dist
```

### Additional Resources / Info

Here are some useful references about GitHub action workflows:

- [Events trigger about GitHub Workflow](https://docs.github.com/en/actions/learn-github-actions/events-that-trigger-workflows)
- [Encrypted secrets settings about GitHub Workflow](https://docs.github.com/en/actions/security-guides/encrypted-secrets)