Mathematics for Safe AI
We don’t yet have known technical solutions to ensure that powerful AI systems interact as intended with real-world systems and populations. A combination of scientific world-models and mathematical proofs may be the answer to ensuring AI provides transformational benefit without harm.
Core beliefs
The core beliefs that underpin this opportunity space:
Future AI systems will be powerful enough to transformatively enhance or threaten human civilisation at a global scale —> we need as-yet-unproven technologies to certify that cyber-physical AI systems will deliver intended benefits while avoiding harms.
Given the potential of AI systems to anticipate and exploit world-states beyond human experience or comprehension, traditional methods of empirical testing will be insufficiently reliable for certification —> mathematical proof offers a critical but underexplored foundation for robust verification of AI.
It will eventually be possible to build mathematically robust, human-auditable models that comprehensively capture the physical phenomena and social affordances that underpin human flourishing —> we should begin developing such world models today to advance transformative AI and provide a basis for provable safety
Observations
Some signposts as to why we see this area as important, underserved, and ripe.
Programme spotlight: Safeguarded AI
As AI becomes more capable, it has the potential to power scientific breakthroughs, enhance global prosperity, and safeguard us from disasters. But only if it’s deployed wisely.
Current techniques working to mitigate the risk of advanced AI systems have serious limitations, as they can’t be relied upon in practice to ensure reliability and safety.
Backed by £59m, this programme looks to combine scientific world models and mathematical proofs ARIA is looking to construct a ‘gatekeeper’ – an AI system designed to understand and reduce the risks of other AI agents. If successful, we’ll unlock the full economic and social benefits of advanced AI systems while minimising risks.
Meet the team
davidad is a software engineer with a multidisciplinary scientific background. He’s spent five years formulating a vision for how mathematical approaches could guarantee reliable and trustworthy AI. Before joining ARIA, davidad co-invented the top-40 cryptocurrency Filecoin and worked as a Senior Software Engineer at Twitter.
"Mathematical proof offers the highest level of assurance known to humankind. Proofs about real-world systems are possible, but take a lot of work. Proofs about AI safety seem intractable now, but soon, AI will be superhuman at writing formal proofs—which simple algorithms can verify with extreme confidence."
Meet the team
Nora is an interdisciplinary researcher with a background spanning complex systems, the philosophy of science, political theory, and AI. Her main interests revolve around making transformative AI go well and understanding intelligent behaviour in natural, social, or artificial systems. Prior to joining ARIA, she co-founded and led PIBBSS, a research initiative exploring new interdisciplinary approaches to tackling important questions in AI risk, governance and safety.
Meet the team
Yasir is an experienced programme manager whose background spans the aviation, tech, emergency services, and defence sectors. Before joining ARIA, he led transformation efforts at Babcock for the London Fire Brigade’s fleet and a global implementation programme at a tech start-up. He supports ARIA as an Operating Partner from Pace.
Explore our other opportunity spaces
Our opportunity spaces are designed as an open invitation for researchers from across disciplines and institutions to learn with us and contribute – a variety of perspectives are just what we need to change what’s possible.