Overview

FactQA is a dynamic short-answer factual question-answering benchmark for measuring hallucinations in large language models. The benchmark is regenerated periodically by the SSPBench pipeline, so every evaluation runs against the latest published version.

Dataset

Generated by the SSPBench factualqa generation engine. The pipeline:

1. Aggregates generated questions across 24 seed topics.
2. Randomly fetches Wiki pages related to each topic.
3. Generates factual questions that the answers can be deducted by the input passages
4. Applies a precomputed quality gate.
5. Filters using RAGAS faithfulness and answer relevancy against the source Wikipedia passage.
6. Estimates multi-model difficulty and separability across test models.
7. Scores per-question novelty versus existing public benchmarks using sentence-transformer embeddings.
8. Measures ranking instability.
9. Runs a multi-objective optimizer (quality, difficulty, novelty, separability) with greedy, diversity-aware selection.
10. Exports the final cut.

Evaluation

This test evaluates a model’s ability to provide short, factual answers or explicit acknowledgment of a lack of knowledge when uncertain. The evaluation process consists of two steps. First, a judge determines whether the response includes an explicit acknowledgment of a lack of information (e.g., "I don't know"). Second, we check the factual correctness of responses. The FactQA Score is calculated as the percentage of responses that are either factually correct or properly acknowledge uncertainty.