Mathematics for Safe AI
We don’t yet have known technical solutions to ensure that powerful AI systems interact as intended with real-world systems and populations. A combination of scientific world-models and mathematical proofs may be the answer to ensuring AI provides transformational benefit without harm.
Opportunity seeds
Outside the scope of programmes and with budgets of up to £500k, these opportunity seeds support ambitious research aligned to the Mathematics for Safe AI opportunity space.
We’re funding 15 opportunity seeds exploring different mathematical approaches to help us verify, understand, and control AI systems.
Formalising and Mitigating the ‘Eliciting Latent Knowledge’ Problem
Francis Rhys Ward, Dr Francis Rhys Ward's Research Group
End-to-End Verification for Constraint Programming
Ciaran McCreesh
SFBench
Jason Gross, Theorem + Redwood Research
Hardware-Level AI Safety Verification
Edoardo Manino, University of Manchester; Mirco Giacobbe, University of Birmingham
Dovetail Research
Alfred Harwood, Dovetail Research
FV-Spec: A Large-Scale Benchmark For Formal Verification of Software
Mike Dodds + Ledah Casburn, Galois
Human Inductive Bias Project
Chris Pang, Meridian
Investigate the feasibility of using LLMs to scalably produce Large Logic-based Expert Systems (LLES)
Joar Skalse
Singular Learning Theory for Safe AI Agents
Daniel Murfet, Timaeus
Learning-theoretic AI Alignment Research Agenda
Vanessa Kosoy, CORAL
Combining Physical and Intentional Stance for Safe AI
Martin Biehl, Cross Labs, Cross Compass; Manuel Baltieri, Araya Inc.; Nathaniel Virgo, University of Hertfordshire
SafePlanBench & Logically Constrained Reinforcement Learning
Agustín Martinez Suñé, University of Oxford
GFlowNet-Steered Probabilistic Program Synthesis for Safer AI
Sam Staton, Nikolay Malkin + Younesse Kaddar, University of Oxford
SCRY-AI: Self-operating Calculations of Risk for Yielding Accurate Insights
Vehbi Deger Turan, Metaculus
Extraction of Structured Knowledge from Language for Scientific Discovery
Nikolay Malkin + Henry Gouk, University of Edinburgh
This opportunity space is part of our rolling seed call experiment – see what's in scope for opportunity seeds in this space by reading the original call for proposals and apply at the link below.