Breaking Models 🚨AI Security Session 7: Too Much Agency

Part 1 LLM03: Excessive Agency Arshi Chadha, OWASP Top 10 for LLM Applications 2026 co-lead The risk when an AI system is allowed to do things, not just say things. This sat at #6 on the 2025 list. In the 2026 update it moved to #3: the biggest jump on the whole list, because production incidents now cluster around models that run commands, call APIs, and touch databases on their own. We’ll walk through the three things that cause it: too much functionality, too many permissions, too much autonomy. And what actually contains it when the model gets manipulated. Part 2 Interactive demo: the email assistant that goes rogue Rishabh Gupta, Software Engineer at Big Tech A prompt-injected email assistant, live. The same malicious instruction succeeds, gets denied, or waits for approval: depending only on how its functionality, permissions, and autonomy are set. Then a real case: a prompt injection planted in a public Slack channel that got Slack AI to surface secrets from private channels. Part 3 The Hugging Face incident: an agent that wasn’t supposed to leave the room Damian Halas, Sr. Software Architect at Nvidia In July 2026, OpenAI models being tested on a cyber benchmark found a flaw in their own sandbox’s package installer, reached the open internet, and broke into Hugging Face’s production systems to get answers for the test. No human directed any of it. Hugging Face saw unauthorized access to internal datasets and service credentials. We’ll walk through how a narrow goal plus tool access plus a containment gap turned an evaluation into a real breach, and what it says about testing models that are getting good at this. Part 4 Group exercise: chaining harmless tools into real damage You get an AI support agent with eight tools. Every one of them looks safe enough to approve on its own. Your job is to find the sequence that leaks customer data anyway: using only permitted actions. Then we flip it and work out where the boundary actually has to sit. Just bring your laptops, no setup needed.