No appeal, no authority: the rule AI systems keep failing

The missing AI feature is due process

The most important AI product feature may not be intelligence. It may be an appeal button that actually works. Today's stories span faculty practice, book markets, lie detection, human-rights monitoring, healthcare, and gig work. Their common question is not whether a model can produce an answer. It is whether an institution can use that answer to close a door while the person on the other side has no practical way to understand, challenge, or escape it.

That distinction separates assistance from authority. A chatbot can suggest. A platform can route. A detector can score. But once a university, employer, clinic, company, or public body turns that output into a consequential decision, the system enters the territory of rights and obligations. Convenience is no longer a sufficient design standard.

A benchmark is not a license to judge

Anthropic's alignment researchers trained lie detectors across 12 settings and roughly 200,000 labeled examples. In familiar conditions, performance rose from an AUROC of 0.60 to 0.95. Across different categories of lies, however, transfer fell to roughly 0.70 to 0.75, and larger prompted models often beat the fine-tuned detectors. About a quarter of labels changed during one cleaning process, underscoring how unstable the supposed ground truth can be.

This is exactly how an impressive laboratory number can become a dangerous institutional shortcut. A detector that recognizes yesterday's lie may misread tomorrow's unfamiliar behavior. The correct response is not to dismiss the research; the team explicitly warns against broad deployment claims. The lesson is that uncertainty must survive the trip from benchmark to decision. A score should trigger investigation, not silently become guilt.

The front door can become a private checkpoint

A Nature Health Perspective argues that consumer AI is moving beyond health information and toward pathway control. Systems are beginning to connect medical records, appointment booking, pharmacy fulfilment, payments, and clinical workflows. That integration may help people complete care, especially where conventional pathways are fragmented. It also concentrates power over how symptoms are interpreted, where patients are routed, and what options they see.

A helpful interface becomes a checkpoint when the user cannot inspect the routing logic, correct bad data, transfer a record, or choose another path. Health governance therefore has to follow the entire pathway, not stop at model accuracy. Procurement rules should require routing transparency, data portability, an accountable clinical escalation, and a safe way to leave the platform without losing continuity of care.

Livelihood decisions require a person with power

The Dutch data-protection authority fined Uber 825 million euros after concluding that automated suspensions significantly affected drivers without adequate information. Uber disputes the decision, says permanent deactivation required human review, and plans to appeal. The legal dispute will continue, but the governance principle should not wait: an affected worker needs more than the existence of a nominal human somewhere in the process.

Meaningful review requires notice of the allegation, access to relevant evidence, enough time to respond, and a reviewer authorized to reverse the outcome. If a person must navigate an opaque support maze while income disappears, the system has automated the consequence and outsourced the appeal back to the victim.

Control of evidence is also control of appeal

Public-interest groups asked the Federal Trade Commission to investigate allegations that leading AI developers bulk-bought books, scanned them for training, and destroyed the physical copies. CBS reports that the groups call it a hoard-and-destroy pipeline and warn that some books could be among the last surviving copies. The allegations require investigation; they should not be treated as an established industry-wide fact.

The underlying danger is larger than inventory. People cannot challenge a model's account of culture, history, or scholarship if the source trail has been concentrated, obscured, or destroyed. AI provenance begins before training. Institutions that turn books, records, and public evidence into private model capability should preserve source integrity and independent access rather than treating the physical record as disposable input.

Public-interest AI needs public-interest constraints

UN News describes AI projects intended to advance human rights by turning fragmented information into usable signals. That is a valuable ambition: evidence arriving faster can help investigators and institutions see patterns that manual review might miss. Yet human-rights work is also where false confidence, missing context, surveillance, and political misuse carry exceptional costs.

A rights-oriented system must preserve source context, protect vulnerable people, disclose uncertainty, and keep legal and moral judgment with accountable humans. The point is not that public institutions should avoid AI. It is that their systems must make contestability a first-class capability rather than a promise added after deployment.

No appeal should mean no automated authority

Universities are already deciding how AI will shape teaching, research, and faculty work. Those choices will teach students something deeper than any individual policy: whether institutional efficiency outranks intellectual accountability. The same lesson travels into workplaces, clinics, courts, and government. If people experience AI as a decision they cannot inspect or challenge, resistance is not ignorance. It is a rational response to unaccountable power.

The rule should be blunt. Any AI-influenced system that can materially alter a person's education, evidence, care, rights, or livelihood must provide notice, provenance, a real human appeal, and a workable exit. If an institution cannot provide those four protections, the machine may advise. It should not decide.

• Notice: tell people when AI materially shaped a consequential recommendation or decision.
• Provenance: preserve the source data, model, instructions, edits, uncertainty, and approval trail.
• Appeal: route challenges to an identified human with evidence, time, and authority to reverse the outcome.
• Exit: let people change providers or processes without losing their education, record, care, income, or access to public services.
• Audit: test unfamiliar cases and real-world failure rates, not only benchmark performance on known distributions.