diverse derived malicious prompts bypass KoalaAI Text-Moderation
Hello KoalaAI Text-Moderation Maintainer,
We are security researchers evaluating whether prompt safety checkers detect malicious prompts derived systematically from blocked originals. The derived prompts use different wording and token patterns while retaining the original harmful intent when submitted to downstream text-to-image generators.
We evaluated KoalaAI Text-Moderation using the same inference configuration and decision rule for original and derived prompts across pornography, illegal activity, disturbing content, hate, and violence. The checker accepted 65.96% of the original malicious prompts and 90.21% of the derived malicious prompts, an increase of 24.25 percentage points. For pornography, acceptance increased from 6.67% to 76.67%. Acceptance increased from 86% to 94% for disturbing content, from 70% to 96% for hate, and from 72.41% to 89.66% for violence.
The derived prompts were not optimized against KoalaAI Text-Moderation and transferred to several independently developed safety checkers. This indicates a generalization weakness rather than a checker-specific query attack. Applications relying on this model as their only prompt-side control may accept transformed malicious inputs that retain harmful downstream image-generation intent.