Dataset · bench-llm
OR-Bench language model over-refusal benchmark
bench-llm/or-bench
A prompt set to check the balance between LLM safety and their tendency to refuse harmless prompts.
What's inside
An English dataset ranging from 10 to 100 thousand examples, including text and visual data (including the OR-Bench-Hard-1K and OR-Bench-Toxic subsets). It measures the model's refusal rate on both genuinely toxic queries and complex yet safe prompts.
What you can do with it
- Test LLMs for excessive censorship and false refusals (over-refusal)
- Calibrate alignment settings and safety filters before model release
- Tasks
- Text generation, Question answering
- Languages
- English
- Size
- 10K–100K rows
- Downloads this month
- 21.7K
- Likes
- 26
Models trained on it
- mradermacher/Spellguard-1.0-8B-GGUF 754
- mradermacher/GPT-Distill-Qwen3-8B-Spellguard-SLERP-GGUF 337
- AxiomicLabs/X3Guard-150MText classification 294
- shesettipavankumarswamy/ahd-jailbreak-v1Text classification 129
- UnderTides/DeepSeek-V4.1-Flash-1017-heretic-lora-devImage text → text 102
- vllm-sr/Decision-1.0-Route-0.6BZero shot classification 101
- Spellguard/Spellguard-1.0-8BText generation 81
- birgermoell/oellm-9b-256k-stage5-e8-search-rl-step160Text generation 51
How to load it
from datasets import load_dataset
ds = load_dataset("bench-llm/or-bench")