Evaluating Agents on Live Gateway Data

Description

Evaluating Agents on Live Gateway Data A half-day, hands-on AI engineering workshop — MelbourneMost eval workshops teach you to evaluate a dataset someone else built. In production, the dataset is whatever your AI gateway captured yesterday — and most teams are throwing all of it away.In this session we turn that around. At the start of the morning we switch on a live fleet of AI agents hitting a real AI gateway (Bifrost) in real time. For the rest of the session, you're building against data that's still arriving — not a static Colab notebook, not toy examples. By the time you leave, you'll have built a golden dataset from real traffic, shipped a working eval suite, and made a ship/no-ship call on a live prompt A/B test — the same decision a production AI team makes every week, compressed into one morning.What you'll doWho it's forAI/ML engineers, data engineers, and product teams who are shipping (or about to ship) AI features and need confidence their system actually works reliably in production. Basic familiarity with AI/LLM concepts is assumed — no agent-building experience required.What you'll leave withA golden dataset seed built from real traffic, a working eval suite you can extend, and a 30-day plan for building this into your own production stack.Logistics📅 Wednesday 22 July 🕤 9:30am – 12:30pm 📍 Stone & Chalk Melbourne, Level 1, 121 King St, Melbourne VIC 3000 ☕ Morning tea provided, courtesy of Soul OriginBring a laptop with Python installed — full setup instructions will be sent ahead of the session.Run by data engineers, for data engineers. Cloud Shuttle · DataEngBytes · cloudshuttle.com.au