Moderating text with the Natural Language API
Tags: googlecloud, nlp, moderation
Published: 2023-06-20 15:46:47

---
title: "Moderating text with the Natural Language API"
published: true
description: "The Natural Language API lets you extract information from unstructured text using Google machine learning and now also supports text moderation."
tags: googlecloud,nlp,moderation
cover_image: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xfskj3rf53a4lr7vcq6n.jpg
canonical_url: https://medium.com/google-cloud/moderating-text-with-the-natural-language-api-5d379727da2c?source=friends_link&sk=59f60f4a09b20db855cab45b3681f6ef
---

> - 2023–09–12: text moderation got Generally Available (GA) over the summer + added link to Sep. blog post
> - Credit: Photo by [Da Nina](https://unsplash.com/fr/@daninasplash) on [Unsplash](https://unsplash.com/photos/MBqwXZTfkdA)

The [Natural Language API](https://cloud.google.com/natural-language/docs/) lets you extract information from unstructured text using Google machine learning and provides a solution to the following problems:

- Sentiment analysis
- Entity analysis
- Entity sentiment analysis
- Syntax analysis
- Content classification
- **Text moderation**

## 🔍 Moderation categories

Text moderation lets you detect sensitive or harmful content. The first moderation category that comes to mind is "toxicity", but there can be many more topics of interest. A [PaLM 2](https://blog.google/technology/ai/google-palm-2-ai-large-language-model)-based model powers the predictions and scores 16 categories:

|            |                       |                   |                |
| ---------- | --------------------- | ----------------- | -------------- |
| Toxic      | Insult                | Public Safety     | War & Conflict |
| Derogatory | Profanity             | Health            | Finance        |
| Violent    | Death, Harm & Tragedy | Religion & Belief | Politics       |
| Sexual     | Firearms & Weapons    | Illicit Drugs     | Legal          |

## ⚡ Moderating text

Like always, you can call the API through the REST/RPC interfaces or with idiomatic client libraries.

Here is an example using the Python client library ([`google-cloud-language`](https://cloud.google.com/python/docs/reference/language)) and the `moderate_text` method:

```python
from google.cloud import language

def moderate_text(text: str) -> language.ModerateTextResponse:
    client = language.LanguageServiceClient()
    document = language.Document(
        content=text,
        type_=language.Document.Type.PLAIN_TEXT,
    )
    return client.moderate_text(document=document)

text = (
    "I have to read Ulysses by James Joyce.\n"
    "I'm a little over halfway through and I hate it.\n"
    "What a pile of garbage!"
)
response = moderate_text(text)
```

> 🚀 It's fast! The model latency is very low, allowing real-time analyses.

The response contains confidence scores for each moderation category. Let's sort them out:

```python
import pandas as pd

def confidence(category: language.ClassificationCategory) -> float:
    return category.confidence

columns = ["category", "confidence"]
categories = sorted(
    response.moderation_categories,
    key=confidence,
    reverse=True,
)
data = ((category.name, category.confidence) for category in categories)
df = pd.DataFrame(columns=columns, data=data)

print(f"Text analyzed:\n{text}\n")
print(f"Moderation categories:\n{df}")
```

You may typically ignore scores below 50% and calibrate your solution by defining upper limits (or buckets) for the confidence scores. In this example, depending on your thresholds, you may flag the text as disrespectful (toxic) and insulting:

```
Text analyzed:
I have to read Ulysses by James Joyce.
I'm a little over halfway through and I hate it.
What a pile of garbage!

Moderation categories:
                 category  confidence
0                   Toxic    0.680873
1                  Insult    0.609475
2               Profanity    0.482516
3                 Violent    0.333333
4                Politics    0.237705
5   Death, Harm & Tragedy    0.189759
6                 Finance    0.176955
7       Religion & Belief    0.151079
8                   Legal    0.100946
9                  Health    0.096305
10          Illicit Drugs    0.083333
11     Firearms & Weapons    0.076923
12             Derogatory    0.073953
13         War & Conflict    0.052632
14          Public Safety    0.051813
15                 Sexual    0.028222
```

## 🖖 More

- Run this Colab notebook: [Using the Natural Language API](https://github.com/GoogleCloudPlatform/devrel-demos/blob/main/other/colab/Using%20the%20Natural%20Language%20API%20with%20Python.ipynb)
- See the [supported languages](https://cloud.google.com/natural-language/docs/languages#text_moderation)
- Read more about [text moderation](https://cloud.google.com/natural-language/docs/moderating-text)
- See the latest blog post on [Improving Trust in AI and Online Communities](https://cloud.google.com/blog/products/ai-machine-learning/google-cloud-text-moderation)
- Follow me on [Twitter](https://twitter.com/PicardParis) or [LinkedIn](https://linkedin.com/in/PicardParis) for more cloud explorations