How we read the signal

Analysis frame

Evidence level

Primary-source evidence

Analytical lens

Separate sustained-interaction testing from first-response refusal rates and from real-world attack outcomes.

Affected groups
  • People exposed to malicious uses of AI
  • Policymakers evaluating safety claims
  • Developers and independent safety testers
What remains unknown
  • Whether current model versions withstand comparable tests
  • How much capability the systems added compared with investigators working without AI
Second-order effects to watch
  • A focus on first-response refusals could reward narrow demonstrations instead of robust safeguards.
  • Repeatable, responsibly reported tests could improve accountability without circulating operational details.

The calendar is part of the finding

The report is dated October 16 and concerns spring testing. Its public appearance on November 18 does not make its model versions current. One test per platform cannot supply a representative failure rate, and validation of portions of a plan is not validation of a completed real-world attack.

Keep the scrutiny, leave out the recipe

We are reporting the governance implication, not reproducing prompts or operational instructions. An evaluator can disclose when and how a claim was tested, the limits of the result and whether remediation was independently retested without exposing details that facilitate harm.

Primary trail

Go to the source

Read the evidence behind this analysis. External links open in a new tab.

GAO — Controlled malicious-use testing, October report ABC — November 18 report and provider responses