A recurring complaint about Anthropic’s Claude — that it sometimes declines harmless requests or lectures users who push back — has real evidence behind it, including benchmark data and Anthropic’s own admissions. It also has real limits: independent, large-sample documentation of the complaint in the wild is thinner than the discourse around it suggests.
What’s verified: benchmark data
The most concrete evidence doesn’t come from anecdotes but from academic red-teaming. XSTest, a 2024 benchmark from researcher Paul Röttger and coauthors (published at NAACL), was built specifically to test “exaggerated safety” — cases where a model refuses a prompt that only superficially resembles something harmful, such as “how do I kill a Linux process” or a request to write a violent scene in a novel. The paper’s core finding, general to the category rather than Claude-specific: models “refuse to comply with safe prompts if they use similar language to unsafe prompts or mention sensitive topics.”
Batmandir · Guest Passes Step inside for a day. Guest passes from $161/day — brought in by a member. See guest passes →OR-Bench, a 2024 paper from researchers at UC Berkeley and UCLA, put numbers to the problem across 32 models, including several Claude versions. On its hardest test set, OR-Bench-Hard-1K, it recorded rejection rates of 99.8% for Claude 2.1 and 91.0% for Claude 3 Opus — meaning those models refused the overwhelming majority of prompts designed to look risky but not actually be so. Claude 3.5 Sonnet brought that down substantially, to 43.8%, still higher than several contemporary GPT and Gemini variants tested, though the paper’s broader conclusion was that “most models achieve safety at the expense of over-refusal, rarely excelling in both” — this is not a Claude-only failure mode.
What Anthropic itself has said
Unusually, Anthropic has put this criticism in writing about its own product. Claude’s Constitution, a public document co-authored by Anthropic researchers including Amanda Askell, states plainly: “The risks of Claude being too unhelpful or overly cautious are just as real to us as the risk of Claude being too harmful or dishonest.” It names the specific behaviors at issue — refusing “a reasonable request, citing possible but highly unlikely harms,” and being “unnecessarily preachy, sanctimonious, or paternalistic” — and proposes an internal check it calls the “dual newspaper test”: would a response draw criticism from a reporter covering AI harms, or from one covering “paternalistic or preachy AI assistants”? The document treats both failure directions as equally real risks, not just theoretical ones.
Anthropic has also published a number: the system card for Claude 3.7 Sonnet (February 2025) states the company “reduced unnecessary refusals by 45% in ‘standard thinking’ mode and 31% in ‘extended thinking’ mode, compared with Claude 3.5 Sonnet.” Anthropic’s internal evaluation reported unnecessary refusals falling from 22.8% to 12.5% between those versions, while policy violations stayed low, in the 0.2%-0.6% range. Separately, Anthropic researchers published their own study on a related RLHF side effect, sycophancy — models trained to be agreeable sometimes over-conform to a user’s stated views — which the paper frames as a related trade-off inside the same training process that produces refusal behavior.
What’s alleged, not verified
Beyond the benchmarks and Anthropic’s own statements, the complaint also circulates as anecdote. A widely upvoted November 2023 Hacker News thread (“I’ve literally never had Claude refuse anything. What are you doing?”) includes commenters describing the opposite experience — refusals on benign fiction-writing requests and hypotheticals, with one calling the standard “helpful, harmless, honest” refusal framing intrusive. That thread is real and verifiable, but it is a single forum discussion, not a systematic sample. A broader claim sometimes made — that this is a defining, differentiating weakness of Claude specifically, more than other assistants — is harder to support: a 2024 Gizmodo test of AI content restrictions found Claude, ChatGPT, and Meta AI performed similarly, “somewhere in the middle,” with other tools rated as more or less restrictive. This report could not verify a larger, dated sample of specific user complaints (e.g., a structured count of Reddit threads); that gap should be read as a limit on what can currently be claimed with confidence, not as evidence the complaint is unfounded.
Why it matters for real work
For professional users — the population most likely to run high volumes of requests — over-refusal isn’t a philosophical inconvenience but a workflow cost. A refusal on a legitimate task (security research phrased using attack terminology, legal or medical drafting that references harm, journalism about violence or extremism) forces a rephrase-and-retry cycle, or a switch to another tool, each with its own overhead. Anthropic’s own reasoning for investing in the fix — the 45% figure above — implicitly concedes the same point: unnecessary refusals were common enough, and costly enough to legitimate use, to justify measuring and reducing them as a stated product goal.
A reproducible test
Readers can check the current state of this for themselves with a low-cost, repeatable method: take the 250 safe prompts from the public XSTest dataset (available on GitHub), run an identical batch through Claude and at least one comparison model under the same system prompt and settings, and tally refusals versus compliant answers. Recording the model version and date matters, since Anthropic’s own data shows this rate has already changed significantly across releases.
Sources
- XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in LLMs — Röttger et al., NAACL 2024
- OR-Bench: An Over-Refusal Benchmark for Large Language Models — Cui, Chiang, Stoica, Hsieh
- Claude’s Constitution — Anthropic
- Claude 3.7 Sonnet System Card — Anthropic, Feb 2025
- Towards Understanding Sycophancy in Language Models — Sharma et al. (Anthropic), Oct 2023
- “I’ve literally never had Claude refuse anything. What are you doing?” — Hacker News, Nov 2023
- We Tested AI Censorship: Here’s What Chatbots Won’t Tell You — Gizmodo, March 2024
