Automate your posting process on DEV with Python, Bash and GitLab CI
Tags: linux, tutorial, bash, gitlab
Published: 2021-10-27 01:23:41

The common process for creating a new post on DEV is:

- Login in
- Click on "Create Post"
- Write the content of your post using Markdown
- Click on "Publish"

## DEV API

You can also use the [DEV API](https://developers.forem.com/api) for creating a new post. According to the documentation, you can create a new post using `curl`:

```bash
curl -X POST -H "Content-Type: application/json" \
 -H "api-key: API_KEY" \
 -d '{"article":{"title":"Title","body_markdown":"Body","published":false,"tags":["discuss", "javascript"]}}' \
 https://dev.to/api/articles
```

In the above command you must replace the value of:

- title
- body_markdown
- published
- tags

according to the details of your post.

`title` is the name of your post.

`body_markdown` is equal to the content of your post. You can create a Markdown file using an editor like [ghostwriter](https://wereturtle.github.io/ghostwriter/) and pass the content of that file to this variable. Create a folder named `articles` to store your Markdown files.

`published` set to `false` or `true`, when set to `false` your post will be saved as draft.

`tags` can be any value [here](https://dev.to/tags) and you can assign more than one tag.

You need to generate an API Key following the instructions [here](https://developers.forem.com/api/#section/Authentication/api_key) and replace that value in the script.

## Bash

If you're passing the content of your post from a Markdown file you must format it first, you can create a Bash script, `publish.sh`, to do it:

```bash
md="articles/article.md" # Markdown file

markdown=$(sed 's/$/\\n/' $md | tr -d '\n')
dev_api_key="API KEY"

curl -X POST -H "Content-Type: application/json" \
 -H "api-key: $dev_api_key" \
 -d '{"article":{"title":"Title","body_markdown":"'"$markdown"'","published":true,"tags":["discuss", "javascript"]}}' \
 https://dev.to/api/articles
```

In your script, you can define a variable with the tags of your post and replace the value of `tags` in the `curl` command, you can modify your script as follows:

```bash
md="articles/article.md" # Markdown file
t=("linux tutorial") # Article tags

markdown=$(sed 's/$/\\n/' $md | tr -d '\n')
dev_api_key="API KEY"

tags="["
for i in "${t[@]}"; do
 tags+="\"$i\""
 if [[ $i != ${t[-1]} ]]; then
 tags+=","
 fi
done
tags+="]"

json='{"article":{"title":"Title","body_markdown": "'"$markdown"'","published":true,"tags":["tag1", "tag2"]}}'
json=$(echo $json | jq --argjson tags $tags '.article.tags |= $tags')
```

The script will get the tags from the `t` variable and create a string similar to `"linux", "tutorial"`, this value is assigned to the `tags` variable.

Then a `json` string is defined and `tags` variable is assigned with temporary value, `["tag1", "tag2"]`, that will be replaced using `jq`. Install `jq` on your system if it isn't installed.

The `curl` command is updated and will look like this:

```bash
curl -X POST -H "Content-Type: application/json" \
 -H "api-key: $dev_api_key" \
 -d "$json" \
 https://dev.to/api/articles
```

Now you can publish a new post by running the Bash script you created before. Using only an offline editor and the command line.

## Python
A YAML file, `articles.yml`, containing details of your post, should be created in the `articles` folder:

```yaml
recent_articles:
 - title: "Title"
   body: "articles/article.md"
   tags: "linux tutorial"
```

`title` is the title of your post.

`body` is the name of the Markdown file

`tags` is the tags assigned to your post

To get these values, a Python script is created, `publish.py`, then this script will run the Bash script previously created and pass those values.

`sed` command is used to escape characters in the Markdown file but some of them are not properly escaped. You can use `json.dumps()` in the Python script to get the JSON string required to pass to the `curl` command.

```python
import yaml
import subprocess
import json

def json_escape(md):
 markdown=json.dumps(md)
 return markdown

with open('articles/articles.yml') as f:
 # use safe_load instead load
 dict = yaml.safe_load(f)

title = dict['recent_articles'][0]['title']
body = dict['recent_articles'][0]['body']
tags = dict['recent_articles'][0]['tags']

f = open('articles/article.md', 'r')
md = f.read()
markdown = json_escape(md)

subprocess.run(["/bin/bash", "script.sh", title, body, tags, markdown])
```

Before running this scipt, you must install the `PyYAML` library by running:

```bash
pip install PyYAML
```

The Bash script must be updated:

```
title=$1 # Article title
md=$2 # Markdown file
t=($3) # Article tags
markdown=$4

dev_api_key="API KEY"

tags="["
for i in "${t[@]}"; do
 tags+="\"$i\""
 if [[ $i != ${t[-1]} ]]; then
 tags+=","
 fi
done
tags+="]"

json='{"article":{"title":"'"$title"'","body_markdown":'$markdown',"published":false,"tags":["tag1", "tag2"]}}'
json=$(echo $json | jq --argjson tags $tags '.article.tags |= $tags')

curl -X POST -H "Content-Type: application/json" \
 -H "api-key: $dev_api_key" \
 -d "$json" \
 https://dev.to/api/articles
```

## GitLab CI
Now the process has changed to this:

- Create your post using a Markdown editor
- Edit `articles.yml` and add details of your post
- Run `publish.py` from the command line

You still have to run the Python script manually and as it would be great to also announce your post on Twitter, you will use GitLab CI to automate part of the process.

Create a GitLab repository and upload the scripts you previously created. Replace `dev_api_key="API KEY"` with `dev_api_key=sys.argv[1]` in your `publish.sh`. And change your `publish.py` script to:

```python
import yaml
import subprocess
import sys
import json

def json_escape(md):
 markdown=json.dumps(md)
 return markdown

dev_api_key = sys.argv[1]

with open('articles/articles.yml') as f:
 dict = yaml.safe_load(f)

title = dict['recent_articles'][0]['title']
body = dict['recent_articles'][0]['body']
tags = dict['recent_articles'][0]['tags']

f = open(body, 'r')
md = f.read()
markdown = json_escape(md)

subprocess.run(["/bin/bash", "publish.sh", title, body, tags, dev_api_key, markdown])
```

Also, replace the Bash script as shown below:

```bash
title=$1 # Article title
md=$2 # Markdown file
t=($3) # Article tags
dev_api_key=$4
markdown=$5

tags="["
for i in "${t[@]}"; do
 tags+="\"$i\""
 if [[ $i != ${t[-1]} ]]; then
 tags+=","
 fi
done 
tags+="]"

json='{"article":{"title":"'"$title"'","body_markdown": '$markdown',"published":true,"tags":["tag1", "tag2"]}}'
json=$(echo $json | jq --argjson tags $tags '.article.tags |= $tags')

url=$(curl -X POST -H "Content-Type: application/json" \
 -H "api-key: $dev_api_key" \
 -d "$json" \
 https://dev.to/api/articles | jq '.url')

echo $url
```

Before configuring GitLab CI, you have to create a Python script that will be run after your post is published for announcing on Twitter your new content.

### Twitter

If you don't have a developer account, you must create one, in order to use TWitter API. You have to [apply](https://developer.twitter.com/en/apply-for-access) for getting access.

Once you're developer account is approved, go to [developer.twitter.com/en/portal/dashboard](https://developer.twitter.com/en/portal/dashboard) and [create a new app](https://developer.twitter.com/en/portal/apps/new), then generate API Key and Secret, and Access Token and Secret for your app.

Create a Python script:

```python
from twitter import *
import sys
import yaml

with open('articles/articles.yml') as f:
 # use safe_load instead load
 dict = yaml.safe_load(f)

title = dict['recent_articles'][0]['title']
url = sys.argv[5]

t = Twitter(auth=OAuth(sys.argv[1], sys.argv[2], sys.argv[3], sys.argv[4]))

update = "New Blog Post: " + title + "\n" + url + "\n #DEVCommunity"

# Update your status
t.statuses.update(status=update)
```

`twitter` and `PyYAML` libraries are required. This script will get details of your post from `articles/articles.yml` and will be run right after your post is created.

To access Twitter API, values of Access Token and Secret, and API Key and Secret will be passed as parameters to this script.

### Configuring GitLab
Before configuring GitLab CI, go to Settings → CI/CD and add the following masked variables:

- ACCESS_TOKEN
- TOKEN_SECRET
- API_KEY
- API_SECRET
- DEV_API_KEY

GitLab CI pipeline will have two stages, `build` and `deploy`, and two jobs, `publish` and `tweet`.

Create `.gitlab-ci.yml` file:

```yaml
stages:
 - build
 - deploy

publish:
 image: python:3.9.7
 stage: build
 script:
 - apt update && apt install -y jq
 - pip install PyYAML 
 - URL=$(python publish.py $DEV_API_KEY)
 - echo "URL=${URL}" >> build.env
 artifacts:
 reports:
 dotenv: build.env
 allow_failure: false

tweet:
 image: python:3.9.7
 stage: deploy
 script:
 - pip install twitter PyYAML
 - python tweet.py $ACCESS_TOKEN $TOKEN_SECRET $API_KEY $API_SECRET $URL
 needs:
 - job: publish
 artifacts: true
```

In the `publish` job, GitLab CI will install `jq` and `PyYAML`, and run `publish.py` with the value of `DEV_API_KEY` as parameter, this variable is defined in the CI/CD configuration.

This is the output you get after creating a new post using `curl`:

```json
{
 "type_of": "article",
 "id": 150589,
 "title": "Byte Sized Episode 2: The Creation of Graph Theory ",
 "description": "The full story of Leonhard Euler and the creation of this fundamental computer science principle, delivered in a few minutes.",
 "cover_image": "https://res.cloudinary.com/practicaldev/image/fetch/s--qgutBUrH--/c_imagga_scale,f_auto,fl_progressive,h_420,q_auto,w_1000/https://thepracticaldev.s3.amazonaws.com/i/88e62fzblbluz1dm7xjf.png",
 "readable_publish_date": "Aug 1",
 "social_image": "https://res.cloudinary.com/practicaldev/image/fetch/s--6wSHHfwd--/c_imagga_scale,f_auto,fl_progressive,h_500,q_auto,w_1000/https://thepracticaldev.s3.amazonaws.com/i/88e62fzblbluz1dm7xjf.png",
 "tag_list": "computerscience, graphtheory, bytesized, history",
 "tags": [
 "computerscience",
 "graphtheory",
 "bytesized",
 "history"
 ],
 "slug": "byte-sized-episode-2-the-creation-of-graph-theory-34g1",
 "path": "/bytesized/byte-sized-episode-2-the-creation-of-graph-theory-34g1",
 "url": "https://dev.to/bytesized/byte-sized-episode-2-the-creation-of-graph-theory-34g1",
 "canonical_url": "https://dev.to/bytesized/byte-sized-episode-2-the-creation-of-graph-theory-34g1",
 "comments_count": 21,
 "positive_reactions_count": 122,
 "public_reactions_count": 322,
 "collection_id": 1693,
 "created_at": "2019-07-31T11:15:06Z",
 "edited_at": null,
 "crossposted_at": null,
 "published_at": "2019-08-01T15:47:54Z",
 "last_comment_at": "2019-08-06T16:48:10Z",
 "published_timestamp": "2019-08-01T15:47:54Z",
 "reading_time_minutes": 15,
 "body_html": "<p>Today's episode of Byte Sized is about Leonhard Euler and the creation of <a href=\"https://en.wikipedia.org/wiki/Graph_theory\">Graph Theory</a>.</p>\n\n<p>For more about how Graph Theory works, check out this video from BaseCS!</p>...\n",
 "body_markdown": "---\r\ntitle: Byte Sized Episode 2: The Creation of Graph Theory \r\npublished: true\r\ndescription: The full story of Leonhard Euler and the creation of this fundamental computer science principle, delivered in a few minutes.\r\ntags: computerscience, graphtheory, bytesized, history\r\ncover_image: https://thepracticaldev.s3.amazonaws.com/i/88e62fzblbluz1dm7xjf.png\r\nseries: Byte Sized Season 1\r\n---\r\n\r\nToday's episode of Byte Sized is about Leonhard Euler and the creation of [Graph Theory](https://en.wikipedia.org/wiki/Graph_theory).\r\n\r\nFor more about how Graph Theory works, check out this video from BaseCS!...",
 "user": {
 "name": "Vaidehi Joshi",
 "username": "vaidehijoshi",
 "twitter_username": "vaidehijoshi",
 "github_username": "vaidehijoshi",
 "website_url": "http://www.vaidehi.com",
 "profile_image": "https://res.cloudinary.com/practicaldev/image/fetch/s--eDGAYAoK--/c_fill,f_auto,fl_progressive,h_640,q_auto,w_640/https://thepracticaldev.s3.amazonaws.com/uploads/user/profile_image/2882/K2evUksb.jpg",
 "profile_image_90": "https://res.cloudinary.com/practicaldev/image/fetch/s--htZnqMn3--/c_fill,f_auto,fl_progressive,h_90,q_auto,w_90/https://thepracticaldev.s3.amazonaws.com/uploads/user/profile_image/2882/K2evUksb.jpg"
 },
 "organization": {
 "name": "Byte Sized",
 "username": "bytesized",
 "slug": "bytesized",
 "profile_image": "https://res.cloudinary.com/practicaldev/image/fetch/s--sq0DrZfn--/c_fill,f_auto,fl_progressive,h_640,q_auto,w_640/https://thepracticaldev.s3.amazonaws.com/uploads/organization/profile_image/865/652f7998-32a8-4fd9-85ca-dd697d2a9ee9.png",
 "profile_image_90": "https://res.cloudinary.com/practicaldev/image/fetch/s--1Pt_ICL---/c_fill,f_auto,fl_progressive,h_90,q_auto,w_90/https://thepracticaldev.s3.amazonaws.com/uploads/organization/profile_image/865/652f7998-32a8-4fd9-85ca-dd697d2a9ee9.png"
 }
}
```

From above JSON, GitLab CI is getting the value of the `url` variable that contains the URL of your new post, and passing that value to next job.

In the `tweet` job, GitLab CI installs `twitter` and `PyYAML`, Python libraries required, and run `tweet.py`.

Your repository is now configured. When you're ready to publish your post, upload the Markdown file and edit `articles/articles.yml` with details of your post, and GitLab CI will publish your post on DEV and tweet about it.

You can check a repository already configured [here](https://gitlab.com/mattdark/dev-blog-posting).