AIImpactLab
9 impact clustersWork & marketsCognition & learningTechnical failuresPrivacySecurityEnvironmentLaw & informationSocial good & healthSystemic risk
All clusters03 / 09
Impact cluster

Technical failure modes

Hallucination, bias, brittleness, model drift, unreliable agents and failures that emerge in real deployment.

Where, how and why do AI systems fail?
Automatically indexed

46 stories in this cluster

View in archive →
Cognition & learningTurkey+2 clusters01

Sungu, Lira and Duckworth, “Generative AI Can Harm Teaching”

In a randomized field experiment across a chain of middle and high schools in Turkey, giving teachers a generative-AI support tool reduced students’ intrinsic motivation by 0.11 standard deviations. Average academic performance did not change, but students taught by lower-performing teachers experienced significant declines in both performance and confidence, showing that a tool that makes lesson preparation easier for teachers does not automatically improve the student experience.

3 min
Work & marketsGlobal+3 clusters02

Liu et al., “Integrating chemical priors and physical laws to mitigate hallucinations in structure-based drug design”

The NUS/Harbin-led team identifies a domain-specific form of generative-AI hallucination: molecular candidates can receive strong predicted binding scores while violating basic chemistry or producing physically impossible atomic arrangements. Its DrugRPG framework incorporates chemical-foundation-model priors and differentiable physical constraints during molecule generation, reducing severe steric clashes by 65.4% relative to the reported state-of-the-art baseline and increasing by 28.6% the share of generated candidates meeting combined potency, stability, and synthetic-feasibility criteria.

2 min
Cognition & learningGlobal+2 clusters03

Tang et al., “Reinforcement learning for treatment decision-making in sepsis: a scoping review”

Reviewing 72 studies of reinforcement-learning systems for sepsis treatment, the authors found that every study was retrospective, 58 studies—80.6%—relied on the same MIMIC critical-care database, and only 10 used private datasets. Although many papers claimed that AI-derived treatment policies outperformed clinicians, variation in how patient states, treatment actions, rewards, and counterfactual outcomes were defined made those comparisons difficult to validate.

2 min
Technical failuresGlobal+2 clusters04

Wood-Charlson et al., “Advancing FAIR data towards comparable, organized, predictive AI-ready data for community validation”

The authors warn that AI systems can amplify stale annotations, incorrect database relationships, inconsistent standards, and weak provenance when they continuously harvest scientific repositories that were designed as comparatively static resources. They extend the FAIR principles with COPE—Comparable, Organized, Predictive, and Engaged—calling for iterative updates, version tracking, uncertainty estimates, machine-actionable standards, and community validation whenever AI-supported analyses generate new knowledge.

2 min
Technical failuresGlobal+1 clusters05

Microsoft, “Least privilege for AI agents: Identity, access, and tool binding”

Microsoft warns that organizations are deploying autonomous, multi-tool agents faster than their identity and authorization systems are evolving to constrain them. Broad permissions and combinations of individually reasonable access rights can allow agents to correlate information across email, files, tickets, and code repositories, creating risks of unauthorized data access, unintended modification or deletion, privilege escalation, and forensic ambiguity about who authorized an action.

2 min
Cognition & learningUnited Kingdom+3 clusters06

Ofqual, “Approach to regulating the use of artificial intelligence in the qualifications sector”

England’s qualifications regulator states that AI may improve assessment design, marking support, invigilation, and operational efficiency, but it identifies accuracy, reliability, confidentiality, bias, fairness, and accountability as unresolved risks in high-stakes assessment. Ofqual explicitly prohibits AI from serving as the sole marker for regulated qualifications, requires meaningful expert human involvement, and warns that undisclosed AI use in coursework can undermine both learning and the validity of awarded grades.

2 min
Cognition & learningGlobal+2 clusters07

Souei et al., “Artificial intelligence in deep brain stimulation for movement disorders: a systematic review and technology readiness assessment”

Researchers reviewed 239 peer-reviewed studies on AI-supported deep-brain stimulation and found a pronounced gap between reported algorithmic performance and clinical readiness. External validation remained rare, evaluations were predominantly retrospective and single-centre, and more than one-quarter of studies used small, high-dimensional datasets with elevated overfitting risk; most systems therefore remained at early-to-intermediate technology-readiness levels.

2 min
Cognition & learningGlobal+3 clusters08

Hu et al., “A scoping review of explainable artificial intelligence for medical multimodal data”

University of Sydney and UC San Diego researchers reviewed 82 studies combining medical imaging, clinical records, and other health-data modalities. They find that most explanations still assign importance to each modality separately and rely on post-hoc techniques that leave the model’s cross-modal reasoning opaque; standardized evaluation was absent from most studies, qualitative assessment predominated, and only a minority provided sufficiently reproducible public code.

2 min
Technical failuresGlobal+3 clusters09

OpenAI, “GPTRed: Unlocking Self-Improvement for Robustness”

OpenAI introduced GPTRed, an internal automated red-teaming model trained through self-play to discover prompt-injection and agentic-system vulnerabilities and generate adversarial training data for production models. In an internal replication of a published prompt-injection challenge, GPTRed succeeded in 84% of novel scenarios versus 13% for human red-teamers; it also compromised a live autonomous vending agent by altering prices, ordering an expensive product at the minimum permitted price, and cancelling another customer’s order.

2 min
Technical failuresAustralia+1 clusters11

Klindt et al., “A unifying framework from neural superposition to sparse interpretable codes”

Researchers from Australian National University, UC Santa Barbara and partner institutions address the problem of neural networks representing more concepts than they have individual neurons, making internal representations difficult to interpret. Their proposed framework combines identifiability theory, sparse coding and behavior-grounded metrics to determine whether extracted model features correspond to meaningful concepts.

2 min
Work & marketsUnited Kingdom+3 clusters12

UK designation of AWS, Google Cloud, Microsoft, and Oracle as Critical Third Parties

The UK Treasury has designated the principal UK or European cloud entities of Amazon Web Services, Google Cloud, Microsoft, and Oracle as the first “critical third parties” subject to direct Bank of England, Prudential Regulation Authority, and Financial Conduct Authority oversight. Regulators state that disruption at one of these highly concentrated providers could simultaneously affect numerous banks, insurers, financial infrastructures, consumers, and markets.

2 min
Cognition & learningGlobal+2 clusters13

Churpek et al., “Early Nephrology Consultation and Acute Kidney Injury in Hospitalized Patients”

University of Chicago and University of Wisconsin researchers randomized 180 hospitalized patients identified by a real-time machine-learning score as being at elevated risk of acute kidney injury. Triggering an early structured nephrology consultation did not significantly reduce peak creatinine changes, acute kidney injury, mortality, readmission, or other major outcomes; many specialist recommendations were not followed by the treating teams.

2 min
Technical failuresUnited Kingdom+3 clusters14

UK DSIT, “Thematic Review and Gap Analysis on AI Security”

The Department for Science, Innovation and Technology published an independent Lancaster University review that mapped 9,109 peer-reviewed AI-security papers from 2021 through January 2026 across 12 lifecycle themes. Despite rapid publication growth, the review identifies major blind spots in formal verification of training data and model-weight integrity, third-party model provenance, the interaction between AI-specific and conventional IT attack surfaces, end-user and shadow-AI risks, and secure retirement or disposal of frontier models.

2 min
Work & marketsGlobal+2 clusters15

Huang et al., “Autonomous biomedical research with an artificial intelligence agent”

The paper introduces Biomni, a general-purpose biomedical agent that can search literature, formulate hypotheses, select datasets and specialized tools, write analytical code, interpret results, and propose subsequent experiments within an integrated workflow. Stanford reports that a prototype is already used by more than 10,000 laboratories; in one example, it processed over 450 wearable-health files and generated plausible findings in 40 minutes, compared with an estimated 60 or more hours of human work.

2 min
Technical failuresGlobal+2 clusters16

OpenAI converts its Bio Bug Bounty into an ongoing frontier-model program

OpenAI expanded its GPT5.5 Bio Bug Bounty into a standing private program focused on finding “universal jailbreaks” capable of defeating predefined biosafety safeguards, beginning with GPT5.6. The maximum reward was doubled from $25,000 to $50,000 for qualifying GPT5.5 or GPT5.6 jailbreaks; GPT5.5 testing ends July 27, after which GPT5.6 becomes the sole model in scope until the program is updated.

2 min
Technical failuresEuropean Union+2 clusters18

EDPB Guidelines 03/2026 on web scraping for generative AI

The European Data Protection Board adopted guidelines clarifying how GDPR applies to web scraping for generative-AI training and fine-tuning. The guidance treats scraping as large-scale automated extraction that often occurs without individuals’ awareness, says GDPR applies when personal data are collected, stored, organized, or retrieved, and emphasizes purpose limitation, transparency, accuracy, source reliability, timestamps, validation, data minimization, and special-category-data limits.

2 min
Technical failuresAustralia+4 clusters20

Microsoft / Mandala, “Unlocking a virtuous cycle: overcoming barriers to AI in Australian energy systems”

Microsoft’s new Australia-focused energy report frames AI as both a driver of electricity demand and a tool for improving grid efficiency, resilience, flexibility, and renewable integration. The report argues that AI could help utilities forecast failures, optimize grid operations, process drone/satellite/sensor data, improve customer service, and unlock latent transmission capacity, but says adoption is constrained by risk aversion, weak regulatory incentives, capital-expenditure bias, siloed data, cybersecurity/privacy concerns, and lack of responsible-AI operating models.

2 min
Technical failuresAustralia+2 clusters21

Australia AI Safety Forum speech

Australia’s Assistant Minister for Science, Technology and the Digital Economy, Andrew Charlton, used a University of Sydney AI Safety Forum speech to frame advanced AI as a “control problem,” citing evidence from the 2026 International AI Safety Report that frontier models show early signs of deception, cheating, and situational awareness. He argued that misalignment becomes a public-safety issue when AI systems draft legislation, screen welfare claims, manage power grids, or otherwise operate inside high-stakes infrastructure.

2 min
Work & marketsUnited Kingdom+5 clusters22

Bank of England Financial Stability Report

The Bank of England’s July 2026 Financial Stability Report is now out, and Reuters reports that the BoE explicitly treats AI as a growing financial-stability risk through two channels: inflated expectations and leveraged investment in AI-related firms, and rising cyber/operational exposure for banks as frontier and agentic AI systems improve. The key line for understanding AI's impact is that AI risk is now being framed not just as “technology risk,” but as a macro-financial vulnerability tied to equity concentration, corporate debt sustainability, opaque financing, correlated leverage, and faster software-update cycles.

2 min
Work & marketsUnited Kingdom+3 clusters24

FCA Mills Review, “AI and the Future of Retail Financial Services”

The UK Financial Conduct Authority published the Mills Review, a 147-page report on AI in retail financial services. It reports that 81% of surveyed firms are adopting AI, that agentic AI is already being piloted or deployed by more than half of industry respondents, and that by 2030 AI may move from back-office support into consumer-facing systems able to recommend, apply, pay, switch products, or take action under preset goals.

2 min
Work & marketsUnited States+4 clusters25

NIST, “2026 Roadmap on Artificial Intelligence and Machine Learning for Smart Manufacturing”

NIST’s roadmap surveys AI/ML applications across industrial analytics, sensing, autonomous systems, additive and laser-based manufacturing, digital twins, robotics, supply-chain/logistics, and sustainable manufacturing, while stressing deployment challenges around industrial big data, interoperability, heterogeneous sensors and control systems, explainability, reliability, safety, and high-stakes operation. The paper’s value is that it treats AI impact as a standards-and-infrastructure problem: the productivity promise depends on data-centric metrology, interoperable systems, safety guardrails, and reliable deployment in physical production environments, not only better models.

2 min
Technical failuresGlobal+2 clusters26

Shen et al., “Generalizable AI predicts immunotherapy outcomes across cancers and treatments”

A Harvard/Broad/MIT-linked team introduced COMPASS, a pan-cancer foundation model that predicts immune-checkpoint-inhibitor response from tumor transcriptomes and interpretable immune concepts. The model was trained on 10,184 tumors across 33 cancer types and reportedly outperformed 22 existing approaches across 16 clinical cohorts covering seven cancers and six immunotherapy agents, with predicted responders showing longer overall survival.

2 min
Technical failuresUnited States+3 clusters27

Reported White House voluntary frontier-model standards

The Financial Times reports that the White House is accelerating voluntary standards with OpenAI, Anthropic, Google, and other frontier-AI firms, potentially setting benchmarks, release timelines, and access rules for advanced models. This remains reported and pending primary confirmation, but it aligns with the June 2 White House executive order and fact sheet directing a voluntary framework for covered frontier models, classified benchmarking for advanced cyber capabilities, and secure early government access for trusted partners.

2 min
Cognition & learningGlobal+3 clusters28

Shi et al., “Physicians and artificial intelligence diverge in evaluating LLMs on real clinical cases”

This multicenter study involved more than 400 physicians across seven specialties and compared human physician evaluation of LLM outputs with AI-agent evaluation configured to mirror physician assessment. AI evaluators were efficient and directionally aligned with physicians, but did not fully capture human clinical judgment and should not replace physician-centered evaluation.

2 min
Work & marketsGlobal+5 clusters31

UN Independent International Scientific Panel on AI preliminary report

The UN’s new independent scientific panel issued its preliminary global AI assessment, warning that AI capability growth is outpacing both scientific understanding and government capacity. The report flags deceptive model behavior, more autonomous “agentic” systems, potential future self-improving AI linked with biotechnology or quantum computing, and misuse risks in cyberattacks, fraud, misinformation, and employment disruption.

2 min
Technical failuresGlobal+1 clusters32

Nature multi-agent scientific-discovery papers

A new Nature News & Views piece highlights two 2026 Nature papers showing AI agents moving from literature support toward hypothesis generation, experiment planning, and data analysis. One paper introduces Robin, a multi-agent system that generated hypotheses, proposed experiments, interpreted results, and identified therapeutic candidates for dry age-related macular degeneration; another introduces Google/DeepMind’s Gemini-based Co-Scientist, with affiliations including Stanford University School of Medicine and Imperial College London, and reports experimentally validated biomedical hypotheses including acute myeloid leukemia drug-repurposing and combination-therapy candidates.

2 min
Technical failuresGlobal+2 clusters33

OpenAI GeneBench-Pro

OpenAI released GeneBench-Pro, a research-level benchmark for testing whether AI agents can reason through ambiguous computational-biology and translational-medicine problems rather than simply answer clean exam-style questions. The benchmark includes 129 expert-created questions across genomics, quantitative biology, pharmacogenomics, and clinical/translational domains; OpenAI reports GPT5.6 Sol reaching 28.7% overall pass rate and 31.5% in Pro mode, while GPT5 scored below 5%.

2 min
Work & marketsGlobal+4 clusters35

Anthropic Economic Index report, “Cadences”

Anthropic’s new Economic Index report updates its labor-impact measurement pipeline for the shift from chat interactions to long-running agentic work in Claude Code and Claude Cowork. The report finds Claude use increasingly follows real-world economic rhythms, classifies concrete outputs across work/personal/coursework contexts, and links survey responses to privacy-preserving usage data from about 9,700 respondents.

2 min
Technical failuresUnited States+3 clusters36

Anthropic Mythos/Fable fallout becomes a live governance case study

Anthropic’s June 12 statement said the U.S. government ordered it to suspend access to Fable 5 and Mythos 5 for foreign nationals, citing national-security concerns around a possible jailbreak, while Anthropic argued the evidence involved a narrow capability also available in other models and warned that applying this standard broadly could halt frontier deployments.

2 min
Technical failuresGlobal+3 clusters37

Tac, Gardner, and Kuhl, “Generative artificial intelligence creates delicious, sustainable, and nutritious burgers”

Stanford researchers used generative AI trained on 2,216 human-designed burger recipes and 146 ingredients, then sampled one million recipes to optimize taste, environmental impact, and nutrition. In a blinded restaurant sensory evaluation with 101 participants, one mushroom-based formulation had an environmental-impact score more than an order of magnitude lower than the Big Mac benchmark, while a bean-based burger nearly doubled the nutritional score and reduced environmental impact by a factor of six.

2 min
Technical failuresGlobal+1 clusters39

Economist Enterprise / Rubrik, “Power without control”

Economist Enterprise research supported by Rubrik reports that 98% of surveyed large organizations operating AI agents have already experienced a disruptive agent-related incident, while two-thirds lack full visibility into agent actions and only 30% have robust, tested rollback capabilities. The report frames agentic-AI failure as a business-continuity problem rather than a narrow IT problem, highlighting regulatory fines, supply-chain disruption, revenue loss, and reputational damage as key consequences.

2 min
Work & marketsGlobal+3 clusters40

OpenAI, “How agents are transforming work”

OpenAI published a new Economic Research item arguing that agentic AI shifts knowledge work from short prompt-response exchanges to delegated, long-horizon tasks. by May 2026, 80.6% of sampled individual users had made at least one Codex request estimated to exceed 30 minutes of human work, 70.2% had made one exceeding one hour, and 25.6% had made one exceeding eight hours; OpenAI also reports Codex becoming the primary AI tool across departments including Legal, Finance, and Recruiting.

2 min
Technical failuresGlobal+1 clusters42

TRUECAM uncertainty-aware cancer-diagnostics framework

Nature Biomedical Engineering published a lung-cancer pathology AI paper introducing TRUECAM, a framework that detects out-of-scope inputs, filters ambiguous regions, and uses conformal prediction to control error rates; the authors report gains in accuracy, robustness, interpretability, data efficiency, and fairness across datasets and foundation models. its significance is less “AI replaces diagnosis” than “AI deployment requires uncertainty, fairness, and error-control layers.”

2 min
Technical failuresGlobal+2 clusters43

OpenAI Daybreak / GPT5.5Cyber

OpenAI announced an expansion of Daybreak and GPT5.5Cyber for trusted defenders, saying Codex Security has scanned more than 30 million commits across over 30,000 codebases and that GPT5.5Cyber scored 85.6 on CyberGym, 39.5% on ExploitGym, and 69.8% on SEC-bench Pro. This is important because the same capability profile supports faster defensive patching while also demonstrating frontier-model competence at vulnerability validation and exploitation.

2 min
Work & marketsGlobal+3 clusters44

Strong et al., “Human-AI Collaboration in Healthcare: A Scoping Review”

This Oxford-led npj Digital Medicine review screened 17,463 records and included 140 empirical studies of human-AI collaboration in healthcare from January 2015 through October 2025. It finds that the evidence base is concentrated in diagnostic interpretation, while triage, therapeutic, administrative, and system-level workflows remain thinner; it also notes that AI benefits depend heavily on task fit, workflow integration, training, and calibrated trust.

2 min
Technical failuresGlobal+3 clusters46

Amazon Nova Premier critical-risk evaluation

Amazon published a technical report evaluating Nova Premier under its Frontier Model Safety Framework, targeting CBRN, offensive cyber operations, and automated AI R&D through automated benchmarks, expert red-teaming, and uplift studies. Amazon says Nova Premier is its most capable multimodal foundation model, with a one-million-token context window that can analyze large codebases, long documents, and video, but concludes that the model remains safe for public release under its stated thresholds.

2 min