PROBE Evaluation Design Jam

The Gap Between Capability and Verified Safety is Widening Agentic AI systems are shipping faster than we can evaluate them. Meanwhile, regulators across the EU, US, Singapore, and China are demanding documented adversarial testing. Most teams still don't have the tooling or methodology to meet that bar. PROBE is a hands-on evaluation design jam where builders and researchers who care about AI safety + security come together to share and co-design evaluation methods. PROBE is designed as a discussion x working session for sharing benchmark datasets, stress-test harnesses, and open-source tooling that the community can reference as they evaluate agents. The goal is to connect with other builders and researchers interested in AI safety and assurance - and create a working group for AI assurance. Curated by AI Safety Node/Ecomonitor (www.aisafetynode.com) and hosted with Ojin(https://ojin.ai/). **** Program and Working Group **** Talks: Human in the Loop x AI Evaluation, Benchmark Exploring Edge Cases in AI Alignment Humans in the Loop x AI Evaluation Trevor Lohrbeer is a co-founder of AI Safety Berlin, a coworking & community hub, for those working to reduce societal risks from AI in Berlin, and an independent AI safety researcher focused on keeping humans in the loop during recursive self-improvement. Prior to going independent, he spent 16 months working with Redwood Research developing AI control evaluations and doing vulnerability research on Claude Code auto mode, and spent 25 years founding and running Internet and software startups. Learn more about his AI safety work here. A Benchmark exploring Edge Cases in AI Alignment Florian Dietz is an AI alignment researcher and consultant. He will share his work on creating benchmark of scenarios that are Out-Of-Distribution (OOD) for current AI systems Florians' Linkedin: https://www.linkedin.com/in/floriandietz/Elective Design Sprint Sessions Safeguard Stress Testing Design Share and work on evaluation design that probe guardrails, test refusal mechanisms and results (if past evaluations are available) Evaluation Tools + Infra Discuss evaluation tools and infrastructure. Share and work on reusable harnesses, metrics pipelines, and reporting tools that make adversarial testing repeatable, not one-off. Schedule 18-19:00: Kick off and guest speaker on evaluation method Speakers from AI Safety Hub in Berlin and independent research firms to discuss evaluation methods and red teaming x agentic harnesses. 19:00 - 20:00: Evaluation design sprint: evaluation design-> implementation sprint 20:00-20:30: Share eval design work through standup and researcher exchangeThis event is open to all. Safety researchers building adversarial benchmarks and agent builders/ML engineers who are exploring eval tooling will find the evening especially relevant. ******* About Ojin******* Ojin AixHaus is the new AI Playground in Berlin. A collaborative community space where scientists, machine learning engineers, founders, creators, and AI enthusiasts come together to build the future of AI. Ojin AixHaus is free to join, open for startups in residency applications and are currently curating its 2026 events calendar. If you have a burning idea for an event, meetup or hackathon - bring your fire, they bring the space! Find out more here: Ojin AixHaus Community Follow Ojin AixHaus on socials: Instagram: https://www.instagram.com/ojin.ai/ Linkedin: https://www.linkedin.com/showcase/ojin-aixhaus/?viewAsMember=true X: https://x.com/Ojin_AI Ecomonitor/AI Safety Node: AI Safety Node, an initiative by Ecomonitor, bring researchers and product developers together for AI safety/security development and community initiatives. Ecomonitor is a research studio focused on complex risks. Read more about our work here: LinkedIn: www.aisafetynode.com https://www.linkedin.com/company/ecomonitor/AI Safety Node Initiative: www.aisafetynode.com