SF Systems: Research to Practice, from Inference Engines to Multimodal Coding

The SF Systems Club is back with a pair of exciting talks from researchers who are bringing cutting-edge science to the real world! Whether you’re a seasoned inference expert or programming languages enthusiast, you will learn a lot at this one. This meetup is generously hosted by LatchBio at their San Francisco office. Food and drinks will be provided! Agenda Talks Shreya Shankar (CMU): Domain Specific LLM Inference for AI-SQL Database users have long wanted to query unstructured text, and LLMs have finally made this feasible. Major vendors now support AI-SQL, which adds AI-powered filters, joins, maps, and classifiers to SQL. To execute AI-SQL queries, databases lower these queries into plans with one LLM call per row, or, for joins, per tuple in a cross product. Even after query optimization, hundreds of thousands, or millions, of calls may remain! In theory, inference engines should be fantastic at this workload, because each call is almost entirely prefill and returns only a few tokens. Surprisingly, general-purpose inference engines take far longer than expected, because their KV caches do not know what the query will reuse, and processing so many requests adds high host overhead. We use these limitations to motivate Quail, an open source query engine for AI-SQL that our lab is building. We describe Quail's planner and execution engine, show preliminary exciting results, and end with some exciting open problems (and, a plea for help from inference experts). Parker Zeigler (UC Berkeley): Interactive, Multimodal Programming Systems The vast majority of programming systems give developers one modality for writing code: editing text. Text editing is remarkably expressive, but for programming domains with rich visual outputs — such as data visualization, graphics, and image processing — it lacks the live feedback and direct manipulation capabilities that shine in tools like Figma, Photoshop, or Blender. How can we design programming systems that unify the best elements of text editing with direct manipulation, natural language, and other input modalities, allowing developers to use the right modality for the moment? This talk will highlight several systems from my PhD research exploring this question, with particular focus on how both old and new ideas from compilers research can transfer effectively to new kinds of interactive, multimodal programming systems.