Moderation policy and thresholds: how ALLOW, REVIEW, BLOCK work

The model scores the content, and the policy decides what to do with that score. Keeping the two apart lets you change the rules without rebuilding the whole system.

Three decisions

ALLOW: the content is fine with enough confidence, so you publish it automatically.

REVIEW: the model is not sure or the content is disputed, so a human decides.

BLOCK: the content breaks the rules with high confidence, so the system stops it.

The principle is simple: automate the obvious, escalate the uncertain.

A confidence score is not a decision

The model returns a number describing how sure it is. The number alone, say 0.73, does not tell a moderator what to do. The policy is the thresholds: the confidence above which the system blocks on its own, the confidence above which it lets content through on its own, and whatever lies between goes to a human.

False alarms and missed violations

Moving a threshold always costs something. A stricter threshold stops more violations but also more honest content. A looser one lets more through, while your team sees fewer false alarms. There is no threshold that suits everyone, so we count both kinds of error on your sample.

How we choose thresholds

We start from team capacity: how many cases per day your moderators can realistically review. Then the cost of a mistake: different for reviews than for medical content. Finally we check the settings on a sample and adjust.

Very short texts carry little information, so it is usually wise to send them to REVIEW rather than decide automatically.

Rule versions

Every decision is stored with the rule version, so after a threshold change you know which version was in force on a given day.

See also