Documentation

Contents

Introduction

Detox is a client-side library that detects toxic text messages using pre-trained AI models. It consists of two components:

  1. The library to be imported into your client, and

  2. The data file that contains weights and settings for the AI models.

After importing the Detox library and loading the data file, initialize the Moderator object. Then, simply pass each chat message through the Moderator to check if the message is toxic. See the Implementation section for further details.

What you do after detecting a toxic message is up to you. We recommend performing recipient-side filtering (i.e. checking and blocking toxic messages when received) as opposed to sender-side blocking (i.e. stopping toxic messages from being sent). This reduces the likelihood of filter evasion, where players intentionally try to get around the filters, and it gives your players the freedom to opt out of the feature should they wish to.

Implementation

The Moderator class is the entry point to Detox and should be initialized when the game client is launched. It has one required argument for initialization: the Detox data file.

While the game is running, Moderator has optional event listeners that can be triggered in reaction to in-game events. These events provide the Moderator with contextual information so it can more accurately identify toxicity in chat. See the sections below on Threat Level and New Player Protection for more information.

When a chat message is sent or received, use the isToxic method to analyze the message and output a Boolean for whether the message is considered toxic.

Loading Client

During game start up, asset initialization, etc.
const Detox = require('./detox.js');
const fs = require('fs');

let detoxData = JSON.parse(fs.readFileSync('.../Detox_Data.json'));
// Initialize Moderator and set base filter strictness
var moderator = new Detox.Moderator(detoxData, Detox.Strictness.STRICT);

Game Events

Player login, objective met, match ended, etc.
// For new players, provide greater protection against toxicity
moderator.newPlayerProtection(true);

// Stricter filter during heightened times (e.g. player death)
moderator.increaseThreat();

Chat Messages

Chat received from friend, team, public, etc.
function messageReceivedCallback(message) {
    let messageToxic = moderator.isToxic(message);
    if (messageToxic) {
        // Do something, e.g. don’t show the message
    }
}

Filter Strictness

The Detox filter can be set at three levels of strictness: LENIENT, STRICT, and VERY_STRICT. A stricter filter identifies a greater number of toxic messages, but at the cost of being more restrictive for players.

  1. LENIENT: Only strongly demeaning messages or profanity-laden text are considered to be toxic. Most suitable for mature games.

  2. STRICT: Insults and aggressive language are considered to be toxic. Suitable for a general audience.

  3. VERY_STRICT: Messages with profanities or negativity are considered to be toxic. Suitable as an overly cautious approach for a young audience.

The strictness level is provided as the second argument when initializing the Moderator using values from the Detox.Strictness class.

// Optional second argument defaults to Detox.Strictness.LENIENT
var lenientModerator = new Detox.Moderator(detoxData);

// Specify strictness level when initializing Moderator
var strictModerator = new Detox.Moderator(detoxData, Detox.Strictness.STRICT);

Threat Level

Set threat level to reflect the imminent threat of players being toxic due to in-game events, such as deaths, losses, and other occurrences that increase tension between players. Threat dynamically changes the filter strictness, allowing you to tighten the reins to proactively stop a flame war and prevent toxicity from compounding.

Starting from level zero, threat can be increased up to level five.

// Increase threat level by one
moderator.increaseThreat();

// Increase threat level by another three, so threat level is four
moderator.increaseThreat(3);

// Decrease threat level by one, threat level back down to three
moderator.decreaseThreat();

// Reset threat level to zero when a match ends or after a certain time
moderator.resetThreat();

New Player Protection (NPP)

Protect new players by increasing the base threat level, which makes the Moderator more sensitive to toxicity. New players are more likely to be socially anxious in game, as they worry about making mistakes and getting critiqued by other players. NPP lets players pick up a new game without the anxiety of receiving hateful messages while still learning the game.

N.B. New player protection should only be enabled when implementing recipient-side filtering (see Introduction).

// Enable new player protection
moderator.newPlayerProtection(true);

// Disable new player protection, e.g. after player completes tutorial
moderator.newPlayerProtection(false);