Nora Ammann

I'm serve as Techncial Specialist to the Safeguarded AI programme - a >£59M R&D programme at UK's Advanced Research and Invention Agency. Safeguarded AI combines scientific world models and mathemtical proofs to develop quantaitive safety guarantees for safety-critical AI systems. Over the course of the ~4-year programme, we seek to demonstrate a general-purpose workflow to leverage frontier AI to accelerate R&D of superhuman-but-task-specific AI systems with quantitative guarantees. Under Programme Director David 'davidad' Dalrymple and Scientific Director Yoshua Bengio, we want to answer the question of how we can leverage the raw potential of highly advanced AI while keeping risks below a societally acceptable level, and building up civilisational resilience. By developing a proof of concept, we intend to establish the viability of a new & complementary pathway for research and development toward safe and transformative AI.

Learn more on our website, or read the Programme Thesis.

I have trying to catalise work towards Flexible Hardware-Enabled Guarantee (flexHEG) mechanisms. These could be added to AI accelerators to enable multilateral, privacy-preserving and trustworthy verification and automated compliance guarantees for agreements regarding the development and use of advanced AI technology. I co-authored a three part report on flexHEG, and colleagues built and illustrated the design of a rapid prototype. I've also helped coordiante a funding round looking to accelerate the prototyping and iterative development of flexHEG-enabled guaranteeable chips.

In January 2025, together with my co-authors, we published a paper discussion the possibility of gradual human disempowerment, which may occur even any sudden 'loss of control' event or discontinuous AI progress. In short, as critical social these systems — like economy, state, culture — become less reliant on human labor and cognition, this will decrease the extent to which humans can explicitly or implicitly align them. As a result, these systems —and the outcomes they produce— might drift further from providing what humans want. Competitive pressures, 'wicked' interactions across systems and scales, will make it systematically difficult to avoid outsourcing critical societal functions to AI. You can find the full paper at: https://gradual-disempowerment.ai/. I am now thinking about what security, verification and cooperation primitives could reduce the risk of gradual disempowerment.

Between 2021 and 2024, I co-founded and directed PIBBSS - Principles of Intelligent Behaviour in Biological and Social Systems. In that period, PIBBSS raised >$2M , supported 6 in-house, long-term research affiliates, ~50 research fellows for 3-month full-time research fellowships, and organized 15+ AI safety research events. Over the years, we built a substantive and vibrant research community, spanning many disciplines, across academia and industry, both inside and outside of AI safety. While I handed over leadership, I continue to support PIBBSS as a member of the board.