Evaluating Agents on Live Gateway Data

Evaluating Agents on Live Gateway Data A half-day, hands-on AI engineering workshop — Melbourne Most eval workshops teach you to evaluate a dataset someone else built. In production, the dataset is whatever your AI gateway captured yesterday — and most teams are throwing all of it away. In this session we turn that around. At the start of the morning we switch on a live fleet of AI agents hitting a real AI gateway (Bifrost) in real time. For the rest of the session, you're building against data that's still arriving — not a static Colab notebook, not toy examples. By the time you leave, you'll have built a golden dataset from real traffic, shipped a working eval suite, and made a ship/no-ship call on a live prompt A/B test — the same decision a production AI team makes every week, compressed into one morning. What you'll do Who it's for AI/ML engineers, data engineers, and product teams who are shipping (or about to ship) AI features and need confidence their system actually works reliably in production. Basic familiarity with AI/LLM concepts is assumed — no agent-building experience required. What you'll leave with A golden dataset seed built from real traffic, a working eval suite you can extend, and a 30-day plan for building this into your own production stack. Logistics 📅 Wednesday 22 July 🕤 9:30am – 12:30pm 📍 Stone & Chalk Melbourne, Level 1, 121 King St, Melbourne VIC 3000 ☕ Morning tea provided, courtesy of Soul Origin Bring a laptop with Python installed — full setup instructions will be sent ahead of the session. Run by data engineers, for data engineers. Cloud Shuttle · DataEngBytes · cloudshuttle.com.au