Overview

SSPBench Safety is a dynamic safety-alignment benchmark that measures whether a model produces unsafe content in response to adversarial prompts. The benchmark is regenerated periodically by the SSPBench pipeline, so every evaluation runs against the latest published version.

Dataset

The SSPBench pipeline grows the benchmark from a set of seed topics by expanding them into adversarial prompts that span the full landscape of safety concerns. It then filters candidates and selects the final cut through a multi-stage process:

  1. Topic-driven generation — for each seed topic, the pipeline asks a generator model to produce candidate adversarial prompts targeting that domain.
  2. Quality gate — each candidate is scored by an LLM judge for suitability, specificity, clarity, and non-triviality; low-quality candidates are dropped.
  3. Faithfulness and relevancy gate (RAGAS) — candidates with weak grounding in the safety domain are filtered out.
  4. Multi-model difficulty estimation — each candidate is run against a panel of base models. Candidates that all models trivially refuse, or that all models trivially comply with, are filtered out in favour of those that better separate stronger from weaker models.
  5. Novelty score — each candidate is embedded and compared against known safety benchmarks (BeaverTails, HarmfulQA, WildGuard, etc.). Candidates that are too similar to existing public prompts are dropped to keep the dataset out-of-distribution.
  6. Ranking instability— leave-one-out analysis estimates how much each candidate would shift the leaderboard ranking if included; candidates that contribute to a more discriminative ranking are preferred.
  7. Multi-objective optimizer — a weighted score combines quality, difficulty, novelty, and separability. A greedy, diversity-aware selector then picks the final cut while enforcing a minimum embedding distance between selected items to prevent near-duplicates.

The result is a small, hard, novel, and diverse sample that targets safety failures across the full topic surface.

Evaluation

A prompt is judged safe if the model refuses or otherwise avoids producing the disallowed content, and unsafe if the model complies with the harmful request. The safe-rate (across all categories and overall) is the model's SSPBench Safety score.