Creating a Github action to detect toxic comments using TensorFlow.js
Tags: tensorflow, javascript, machinelearning
Published: 2020-08-16 20:37:51

---
title: Creating a Github action to detect toxic comments using TensorFlow.js
published: true
description: Over the weekend, I spent a few hours building a Github action to automatically detect potentially toxic comments and PR reviews.
tags: tensorflowjs, javascript, tensorflow, machinelearning
---

*Post originally posted on my [personal site](https://charliegerard.dev/)*

Over the weekend, I spent a few hours building a Github action to automatically detect potentially toxic comments and PR reviews.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/r2kynkd05wrrzrj19oe5.png)

It uses TensorFlow.js and its [toxicity pre-trained model](https://github.com/tensorflow/tfjs-models/tree/master/toxicity) to assess the level of toxicity, based on the following 7 categories:

- Identity attack
- Insult
- Obscene
- Severe toxicity
- Sexual explicit
- Threat
- Toxicity

When a user posts a new comment or reviews a PR, the action is triggered. If there is a high probability that the content would be classified as toxic, the bot creates a comment tagging the author and advising to update the content.

Here's a quick demo:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/wvbb5y79x4w7toov8ub9.gif)

[repo](https://github.com/charliegerard/safe-space)

## Setup

Before we dive into the code, it is probably important to note that this is a **JavaScript action**. I read that actions could also be in Docker containers but for simplicity, I stuck with JS.

First, I created a action.yml file at the root of my project folder.

Inside this file, I wrote the following code:

```yml
name: "Safe space"
description: "Detect the potential toxicity of PR comments"
inputs:
  GITHUB_TOKEN:
    required: true
  message:
    required: false
  toxicity_threshold:
    required: false
runs:
  using: "node12"
  main: "dist/index.js"
```

The first couple of lines are self-explanatory. Then, the `inputs` property contains 3 different elements.

- The `GITHUB_TOKEN` is a secret token required to authenticate in your workflow run and is automatically generated.
- The `message` property is optional and can be used by people if they want to customise the content of the comment posted by the bot if the action detects toxic comments.
- The `toxicity_threshold` property is also optional and allows people to set a custom threshold that will be used by the machine learning model when making predictions about a comment.

Finally, the settings under `runs` indicate the version of Node.js we want our action to run with, as well as the file the action code lives in.

## Action code

To create a JavaScript action, you need to install and require at least 2 Node.js modules: `@actions/core` and `@actions/github`. As this particular action uses a TensorFlow.js model, I also installed and required `@tensorflow-models/toxicity` and `@tensorflow/tfjs`.

Then, in my `dist/index.js` file, I started writing my action code.

The core setup could look something like this:

```javascript
async function run() {
  const tf = require("@tensorflow/tfjs");
  const toxicity = require("@tensorflow-models/toxicity");
  await tf.setBackend("cpu");

  try {
    const githubToken = core.getInput("GITHUB_TOKEN");
    const customMessage = core.getInput("message");
    const toxicityThreshold = core.getInput("toxicity_threshold");
    const { context } = github;
  } catch (error) {
    core.setFailed(error.message);
  }
}

run();
```

There is a main `run` function that requires the packages needed and sets the backend for TensorFlow.js. Then in a `try`/`catch` statement, the code gets the 3 parameters mentioned a bit earlier that we'll be using soon.
Finally, we get the context of the event when the action is triggered.

### Creating a bot comment when a user posts a comment on an issue or PR

A few different events can trigger a Github action. As this action is interested in getting the comments posted on an issue or PR, we need to start by looking at the payload of the event and see if the property `comment` is defined. Then we can also look at the type of action (here `created` and `edited`), to run the predictions only when a new comment is added or one is edited, but not deleted for example.

More details are available on the [official Github documentation](https://docs.github.com/en/actions).

I then access a few parameters needed to request the comments for the right issue or PR, load the machine learning model, and if the property `match` is true on one of the results coming back from the predictions, it means the comment has been classified as toxic and I generate a new comment with the warning message.

```javascript
if (context.payload.comment) {
  if (
    context.payload.action === "created" ||
    context.payoad.action === "edited"
  ) {
    const issueNumber = context.payload.issue.number;
    const repository = context.payload.repository;
    const octokit = new github.GitHub(githubToken);
    const threshold = toxicityThreshold ? toxicityThreshold : 0.9;
    const model = await toxicity.load(threshold);
    const comments = [];
    const commentsObjects = [];
    const latestComment = [context.payload.comment.body];
    const latestCommentObject = context.payload.comment;
    let toxicComment = undefined;

    model.classify(latestComment).then((predictions) => {
      predictions.forEach((prediction) => {
        if (toxicComment) {
          return;
        }
        prediction.results.forEach((result, index) => {
          if (toxicComment) {
            return;
          }
          if (result.match) {
            const commentAuthor = latestCommentObject.user.login;
            toxicComment = latestComment;
            const message = customMessage
              ? customMessage
              : `<img src="https://media.giphy.com/media/3ohzdQ1IynzclJldUQ/giphy.gif" width="400"/> </br>
                                      Hey @${commentAuthor}! 👋 <br/> PRs and issues should be safe environments but your comment: <strong>"${toxicComment}"</strong> was classified as potentially toxic! 😔</br>
                                      Please consider spending a few seconds editing it and feel free to delete me afterwards! 🙂`;

            return octokit.issues.createComment({
              owner: repository.owner.login,
              repo: repository.name,
              issue_number: issueNumber,
              body: message,
            });
          }
        });
      });
    });
  }
}
```

### Creating a bot comment when a user submits a PR review

The code to run checks on PR reviews is very similar, the main difference is in the first couple of lines. Instead of looking for the `comment` property on the payload, we look for `review`, and the action I am interested in is `submitted`.

```javascript
if (context.payload.review) {
  if (context.payload.action === "submitted") {
    const issueNumber = context.payload.pull_request.number;
    const repository = context.payload.repository;
    const octokit = new github.GitHub(githubToken);
    const threshold = toxicityThreshold ? toxicityThreshold : 0.9;
    const model = await toxicity.load(threshold);
    const reviewComment = [context.payload.review.body];
    const reviewObject = context.payload.review;
    let toxicComment = undefined;
    model.classify(reviewComment).then((predictions) => {
      predictions.forEach((prediction) => {
        if (toxicComment) {
          return;
        }
        prediction.results.forEach((result, index) => {
          if (toxicComment) {
            return;
          }
          if (result.match) {
            const commentAuthor = reviewObject.user.login;
            toxicComment = reviewComment[0];
            const message = customMessage
              ? customMessage
              : `<img src="https://media.giphy.com/media/3ohzdQ1IynzclJldUQ/giphy.gif" width="400"/> </br>
                                      Hey @${commentAuthor}! 👋 <br/> PRs and issues should be safe environments but your comment: <strong>"${toxicComment}"</strong> was classified as potentially toxic! 😔</br>
                                      Please consider spending a few seconds editing it and feel free to delete me afterwards! 🙂`;

            return octokit.issues.createComment({
              owner: repository.owner.login,
              repo: repository.name,
              issue_number: issueNumber,
              body: message,
            });
          }
        });
      });
    });
  }
}
```

## Using the action

To use an action in a repository, we need to create a workflow file.

First, the repository needs to have a `.github` folder with a `workflows` folder inside. Then, we can add a new `.yml` file with the details for the action we want to run.

```yml
on: [issue_comment, pull_request_review]

jobs:
  toxic_check:
    runs-on: ubuntu-latest
    name: Safe space
    steps:
      - uses: actions/checkout@v2
      - name: Safe space - action step
        uses: charliegerard/safe-space@master
        with:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
```

In this code sample, we indicate that we want to trigger this action when events happen around comments in an issue and when a pull request review event happens.
Then, we add that we need to start by using the default `actions/checkout@v2` action and finally, add this toxicity classification action, with some additional parameters, including the required `GITHUB_TOKEN` one.

If you wanted to use the optional properties `message` and `toxicity_threshold`, you could do so like this:

```yml
on: [issue_comment, pull_request_review]

jobs:
  toxic_check:
    runs-on: ubuntu-latest
    name: Safe space
    steps:
      - uses: actions/checkout@v2
      - name: Safe space - action step
        uses: charliegerard/safe-space@master
        with:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          message: "Hello"
          toxicity_threshold: 0.7
```

If you are developing your own action, you can test it by changing the line

```yml
uses: charliegerard/safe-space@master
```

to

```yml
uses: ./
```

One important thing to note, if you want to build your own Github action, is that when using the action type `issue_comment` and `pull_request_review`, you need to push your code to your main (often called "master") branch before being able to test that your code is working in another branch. If you develop everything in a separate branch, the action will not be triggered when writing a comment or reviewing a PR.

And that's it! 🎉

## Potential improvements

At the moment, I invite the user to delete manually the note from the bot once they've updated the content of their toxic comment, however, I think this could be done automatically on edit. When the user edits a comment, I could run the check again and if it is predicted as safe, remove automatically the bot comment so users don't have to do it.