Moderate text

Text moderation analyzes a document against a list of safety attributes, which include "harmful categories" and topics that may be considered sensitive. To moderate the text in a document, call the moderateText method.

A complete list of categories returned for the moderateText method are found here:

Safety attribute confidence scores

Each safety attribute has an associated confidence score between 0.00 and 1.00, reflecting the likelihood of the input or response belonging to a given category.

Sample response

{
  "moderationCategories": [
    {
      "name": "Toxic",
      "confidence": 0.10
    },
    {
      "name": "Insult",
      "confidence": 0.12
    },
    {
      "name": "Profanity",
      "confidence": 0.07
    },
    {
      "name": "Derogatory",
      "confidence": 0.04
    },
    {
      "name": "Sexual",
      "confidence": 0.00
    },
    {
      "name": "Death, Harm & Tragedy",
      "confidence": 0.00
    },
    {
      "name": "Violent",
      "confidence": 0.00
    },
    {
      "name": "Firearms & Weapons",
      "confidence": 0.00
    },
    {
      "name": "Public Safety",
      "confidence": 0.01
    },
    {
      "name": "Health",
      "confidence": 0.01
    },
    {
      "name": "Religion & Belief",
      "confidence": 0.00
    },
    {
      "name": "Illicit Drugs",
      "confidence": 0.01
    },
    {
      "name": "War & Conflict",
      "confidence": 0.02
    },
    {
      "name": "Politics",
      "confidence": 0.01
    },
    {
      "name": "Finance",
      "confidence": 0.00
    },
    {
      "name": "Legal",
      "confidence": 0.00
    }
  ]
}

Test your confidence thresholds

You can test Google's safety filters and define confidence thresholds that are right for your business. By using these thresholds, you can take comprehensive measures to detect content that violates Google's usage policies or terms of service and take appropriate action.

The confidence scores are only predictions. You shouldn't depend on the scores for reliability or accuracy. Google is not responsible for interpreting or using these scores for business decisions.

Difference between probability and severity

The confidence scores indicate that the content belongs to the specified category and not the severity. This is important to consider because some content can have low probability of being unsafe even though the severity of harm could still be high. For example, comparing the sentences:

1. The robot punched me.
2. The robot slashed me up.

Sentence 1 might cause a higher probability of being unsafe but you might consider sentence 2 to be a higher severity in terms of violence.

Therefore, it is important for you to carefully test and consider what the appropriate level of blocking is for your use cases while minimizing harm to end users.

Language support

How to moderate text

This section demonstrates how to moderate text in a document. You need to submit a separate request for each document.

Here is an example of moderating text provided as a string:

curl -X POST \
     -H "Authorization: Bearer "$(gcloud auth application-default print-access-token) \
     -H "Content-Type: application/json; charset=utf-8" \
     --data "{
  'document':{
    'type':'PLAIN_TEXT',
    'content':'Shut up!'
  }
}" "https://language.googleapis.com/v1/documents:moderateText"

Moderate text from Cloud Storage

Here is an example of moderating text stored in a text file on Cloud Storage:

curl -X POST \
     -H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
     -H "Content-Type: application/json; charset=utf-8" \
 https://language.googleapis.com/v1/documents:moderateText -d "{
  'document':{
    'type':'PLAIN_TEXT',
    'gcsContentUri':'gs://<bucket-name>/<object-name>'
  }
}"