Moderation policy and thresholds: how ALLOW, REVIEW, BLOCK work
The model scores the content, and the policy decides what to do with that score. Keeping the two apart lets you change the rules without rebuilding the whole system.
- ALLOW publishes, REVIEW sends to a human, BLOCK stops the content.
- You set thresholds to your team capacity and the cost of a mistake.
- Changing a threshold does not require retraining the model.
Three decisions
ALLOW: the content is fine with enough confidence, so you publish it automatically.
REVIEW: the model is not sure or the content is disputed, so a human decides.
BLOCK: the content breaks the rules with high confidence, so the system stops it.
The principle is simple: automate the obvious, escalate the uncertain.
A confidence score is not a decision
The model returns a number describing how sure it is. The number alone, say 0.73, does not tell a moderator what to do. The policy is the thresholds: the confidence above which the system blocks on its own, the confidence above which it lets content through on its own, and whatever lies between goes to a human.
False alarms and missed violations
Moving a threshold always costs something. A stricter threshold stops more violations but also more honest content. A looser one lets more through, while your team sees fewer false alarms. There is no threshold that suits everyone, so we count both kinds of error on your sample.
How we choose thresholds
We start from team capacity: how many cases per day your moderators can realistically review. Then the cost of a mistake: different for reviews than for medical content. Finally we check the settings on a sample and adjust.
Very short texts carry little information, so it is usually wise to send them to REVIEW rather than decide automatically.
Rule versions
Every decision is stored with the rule version, so after a threshold change you know which version was in force on a given day.