Mistral AI announced yesterday (August 4) the launch of Shieldstral, an AI model for content moderation with a total of 3B (3 billion) parameters. It uses open weights and is released under the Apache 2.0 license.
The model is now available on Hugging Face, supports 12 languages, and can run on a single 16GB GPU. Mistral stated that the model delivers content safety performance comparable to that of open models seven times its size, and achieves SOTA in multimodal content moderation.
Note: SOTA stands for state-of-the-art. It refers to the set of models or algorithms that perform best and represent the most advanced level in a specific artificial intelligence task or benchmark. It continually changes as technology evolves and new research is published.
Mistral pointed out that most guard models hard-code the taxonomy of harmful categories into the model weights. When the product is adapted to a different use case, developers typically need to retrain the model.
Shieldstral instead places moderation policies in the input: operators can enter a “yes or no” question, then provide details about the evaluation scenario and strictness level. The model subsequently generates a calibrated safety score from the individual output tokens.
The model converts each moderation task into a binary question-and-answer task, with input containing 3 labeled fields:
<Instruct>: Describes the evaluation scenario and strictness level.
<Query>: Asks a single “yes or no” question, such as “Does this content promote physical violence?”
<Document>: Provides the content to be moderated. This can be a prompt, a response, a combination of a prompt and response, or an image with optional accompanying text.
During inference, the model reads only the logit values of the two tokens “yes” and “no,” then normalizes them into a continuous score using softmax and outputs a binary judgment with 0.5 as the threshold. This mechanism can cover prompt classification, response moderation, refusal detection, and toxicity detection.
