How we read the signal

Analysis frame

Evidence level

Peer-reviewed research

Analytical lens

Comparing how two independent teams define coverage reveals how aggregate numbers can hide absent cases without implying a shared population.

Affected groups
  • Clinics evaluating dermatology software for children and adults
  • Teams evaluating language models across legal contexts
What remains unknown
  • Error rates for any specific clinic or patient group
  • How often the evaluated responses would cause a real-world harm
Second-order effects to watch
  • Purchasers can request subgroup evidence before a deployment decision.
  • Benchmark designers may prioritize new situations over additional near-duplicate examples.

How to request a better evaluation

Ask for sample counts and how missing labels were handled before interpreting a subgroup score. In a legal-safety test, ask who chose the cases, which jurisdictions were covered and exactly what counted as a safe response.

Primary trail

Go to the source

Read the evidence behind this analysis. External links open in a new tab.

Nature Communications — case-grounded legal safety evaluation Nature Communications — public dermatology image collections