Content moderation

​

Why moderation

​

Modalities

​

Prompt moderation

• Input: text field
• Pipeline: a blocklist scan followed by an LLM prohibition check (your account administrator can enable additional optional steps)
• Output: is_prohibited verdict, plus a reason when a prompt is rejected

​

Chat moderation

• Input: text field
• Pipeline: an LLM prohibition check (no blocklist scan)
• Output: is_prohibited verdict, plus a reason when a turn is rejected

​

Image moderation

• Input: image field — a public HTTP/HTTPS URL (direct file upload is currently paused)
• Pipeline: VLM classification + NSFW classification + optional custom VLM children, all in parallel
• Output: is_prohibited verdict and flagged categories

​

Video moderation

• Input: video field — a public HTTP/HTTPS URL up to 300 MB (direct file upload is currently paused)
• Pipeline: the video is classified directly by the vision model (no frame extraction)
• Output: is_prohibited verdict and an informational nsfw label

​

Coming soon

• Audio moderation

​

Asynchronous processing

1. Poll the job status endpoint for results.
2. Stream real-time updates via SSE.
3. Receive results via webhook when the job completes.

​

Pipeline architecture

​

Prompt moderation pipeline

PROMPT_MODERATION (parent)
└── TEXT_MODERATION          — Blocklist scan, LLM prohibition check

​

Chat moderation pipeline

CHAT_MODERATION (parent)
└── CHAT_LLM_CHECK           — LLM prohibition check (no blocklist)

​

Image moderation pipeline

IMAGE_MODERATION (parent)
├── VLM_CLASSIFICATION       — Visual analysis via LLM (one per enabled VLM category)
├── NSFW_CLASSIFICATION       — GPU-based NSFW detection
└── VLM_CLASSIFICATION        — Per-client custom categories (one per enabled custom category)

​

Video moderation pipeline

VIDEO_MODERATION (parent)
├── VLM_CLASSIFICATION       — Video analysis via vision model (AI Adult General, always on)
└── VLM_CLASSIFICATION       — NSFW label (when enabled)

​

Video frame moderation pipeline (legacy)

VIDEO_FRAME_MODERATION (parent)
└── FRAME_EXTRACTION         — Sample frames from video via FFmpeg
    └── FRAME_MODERATION     — Per-frame content analysis (one per frame)