Overview

HelpfulnessBench measures whether a model unnecessarily refuses benign requests that are adjacent to sensitive topics. A safety-tuned model that refuses any question mentioning violence, illness, or politics may look safe on safety benchmarks, but it becomes unhelpful in practice. HelpfulnessBench highlights this trade-off.

Dataset

The SSPBench over-refusal pipeline grows the benchmark from a set of seed topics — domains where over-refusal is most likely (Violence, Hate, Illegal Activities, Sexual Content, etc.) — and produces prompts that touch the sensitive domain but ask for something legitimate: historical analysis, clinical or educational information, abstract discussion, legal context, etc.

The pipeline then filters and selects the final cut through a multi-stage process:

  1. Topic-driven generation — for each seed topic, the pipeline asks a generator model to produce candidate borderline-safe prompts that touch the sensitive domain while staying squarely on the safe side.
  2. Quality gate — each candidate is scored by an LLM judge on suitability, specificity, clarity, and non-triviality; low-quality candidates are dropped.
  3. Multi-model difficulty estimation — every candidate is run against a panel of base models. Candidates that all models comfortably answer, or that all models refuse, are filtered out in favour of those that separate well-calibrated models from over-refusing ones.
  4. Novelty score — each candidate is embedded and compared against known over-refusal benchmarks (XSTest, ORBench, OKTest, WildGuard). Candidates too similar to existing public prompts are dropped, keeping the dataset out-of-distribution.
  5. Ranking instability — leave-one-out analysis estimates how much each candidate would shift the leaderboard ranking if included; candidates that contribute to a more discriminative ranking are preferred.
  6. Multi-objective optimizer — a weighted score combines quality, difficulty, novelty, and separability. A greedy diversity-aware selector then picks the final cut while enforcing a minimum embedding distance between selected items, preventing near-duplicates.

The result is a small, hard, novel, diverse sample that targets over-refusal across the full sensitive-topic surface. Each prompt is tagged with the category that motivated its generation:

Code Category What it probes
DM Deception & Misinformation Prompts that touch on deceptive or misleading content, where an unwarranted refusal is harmful.
HB Harassment & Bullying Prompts adjacent to harassment topics, where benign requests are often over-refused.
HD Hate & Discrimination Prompts that touch on hate speech, where the assistant should still answer legitimate questions.
IA Illegal Activities Prompts adjacent to illegal topics (e.g., historical analysis), where over-refusal would be unhelpful.
PS Privacy & Surveillance Prompts that touch on privacy/surveillance, where benign queries should still be answered.
SP Sensitive Historical/Political Prompts on sensitive historical or political topics that warrant substantive engagement.
SC Sexual Content Prompts adjacent to sexual-content topics, where benign clinical/educational requests should be answered.
UB Unethical Behavior Prompts that touch on unethical behaviour, where the model should not refuse an abstract discussion.
VH Violence & Harm Prompts adjacent to violence/harm topics, where legitimate questions should still receive answers.
OT Other Prompts that do not fall under any of the named parent categories.

This breakdown shows where a model over-refuses. For example, a model that is helpful on Health & Nutrition but refuses every prompt that mentions weapons or politics has a specific calibration problem—not a general one.

Evaluation

For each prompt the model under test generates a response. An LLM compliance judge (gpt-oss-safeguard model, configured with aiXamine's over-refusal policy) then reads the prompt + response and decides whether the model engaged with the request or refused it.

Note the inverted polarity vs SSPBench Safety: here, refusal is the failure mode. The same judge model is used so verdicts are comparable across the safety and over-refusal benchmarks, but the policy the judge is given differs — a refusal that scores well on safety is what scores poorly on helpfulness.