From Pixels to Actions

This talk explores how to design agents that can perceive, reason, and act across diverse digital and physical environments. First, Aravind will present Fara, the latest computer use agent from Microsoft that sets a new SoTA for its size class. Fara adopts a "pixel-in, action-out" formulation: it perceives the computer screen directly through raw screenshots, formulates intermediate reasoning steps, and predicts atomic low-level actions (clicks, scrolls, keystrokes). This results in robust and human-like behaviours that generalise to the fat tail of web tasks. Next, turning from navigating web pages to navigating the physical world. Aravind will later present OpenEQA, a benchmark that requires physical agents to explore physical spaces like homes and offices We'll see that frontier models perform far below humans here, especially on spatial understanding. This motivates Locate3D, a self-supervised model that grounds language directly in raw sensor streams to localise objects in 3D, building the kind of persistent world representations that support memory and navigation. About the speaker: Aravind Rajeswaran is a researcher at Microsoft AI Frontiers, working on reinforcement learning and world models to build generalist AI agents. Before Microsoft, Aravind spent 4 years at Meta (FAIR) working on Embodied (Physical) AI. Some of his notable projects include Fara, OpenEQA, R3M, Decision Transformer, and DAPG. He earned his PhD from the University of Washington with Profs. Sham Kakade and Emo Todorov, and was a visiting postdoc at UC Berkeley with Prof. Pieter Abbeel. He has also received several notable awards including a JP Morgan PhD fellowship in AI, a best paper award from IEEE SIMPAR, and the best undergraduate thesis award from IIT Madras. To attend online: Looking forward to seeing you!