🤖🍨 Sundae Robotics 06: V-JEPA 2 & Predicting Physical Intelligence

🤖🍨 Grab a sundae and join Sundae Robotics, a private, invite-only Sunday series bringing together robotics researchers, founders, and builders working at the frontier of physical intelligence. Sundae Robotics 06From V-JEPA 2 to Physical Intelligence — Prediction, Planning & Post-TrainingFeatured Talk: Predicting Physical Intelligence — Daniel Dugas V-JEPA 2, Predictive World Models & Physical Intelligence Why is intelligence so predictive? And how do we build AI systems that understand physical reality, anticipate what happens next, and use those predictions to act? This session explores V-JEPA 2, Meta FAIR’s self-supervised video world model for visual understanding, prediction, and robot planning, as a starting point for a broader discussion about predictive physical intelligence. Rather than training a model to reconstruct every pixel of a video, V-JEPA 2 learns to predict abstract representations of how scenes evolve. The goal is to capture higher-level, predictable structure—objects, motion, geometry, interactions, and changes in physical state—without spending model capacity reproducing every visual detail. This raises a fundamental question: what should a useful world model actually predict? V-JEPA 2 is pre-trained on large-scale natural video through self-supervised learning, allowing the model to learn broad regularities about how the physical world evolves from passive visual experience. The model can then be adapted with comparatively limited robot interaction data, connecting these learned physical representations to action. The broader scaling hypothesis is compelling: rather than teaching robots the structure of the physical world entirely through expensive embodied experience, we may be able to learn broad physical priors from abundant human and internet video—and use robot data primarily to teach the model how its own actions change the world. A particularly interesting part of V-JEPA 2 is its extension from passive video prediction to action-conditioned robot planning. Instead of asking only: What happens next? a robot world model can ask: What happens next if I take this action? Given a current observation, candidate actions, and a desired goal, the system can reason over possible future states and select actions that move the environment toward that target. This enables behaviors such as reaching, grasping, manipulating previously unseen objects, and pick-and-place using goal images as task specifications. The work points toward a broader vision of world models: AI systems that do more than recognize what is currently visible. They build internal representations of how the world behaves, predict possible futures, evaluate those futures, and use predictions to choose actions. Daniel will draw on his previous work at Meta FAIR to discuss what emerges as predictive world models scale—from features versus pixels, to cross-embodiment representations, latent prediction, and robot planning. From there, we’ll move beyond V-JEPA 2 to a larger question: Once a model can predict the future, what are those predictions actually useful for? We’ll explore a possible decomposition of physical intelligence into three interacting components: Prediction → Value → Action Prediction tells us what might happen.Value tells us which outcomes are desirable.Action determines how to make those outcomes happen. We’ll connect this framing to planning in learned world models, including approaches such as WorldPlanner, which combines action-conditioned visual prediction with Monte Carlo Tree Search and Model Predictive Control. Finally, we’ll look beyond pre-training toward the emerging challenge of Physical AI post-training. As scaling recipes for frontier AI pre-training become increasingly established, roboticists are again confronting questions around reinforcement learning, demonstrations, environment interaction, continual learning, and the role of simulation. The elephant in the room: Is simulation undeniably necessary, utterly hopeless, or both? We’ll discuss how world models, simulation, reinforcement learning, and real-world robot experience might fit together—and whether Physical AI will eventually develop a post-training paradigm analogous to the one emerging for frontier AI agents. Pre-Reading • V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planninghttps://arxiv.org/abs/2506.09985 • Meta AI — V-JEPA 2https://ai.meta.com/research/vjepa/ • Code + Models — V-JEPA 2https://github.com/facebookresearch/vjepa2 • The Paths Perspective on Value Learninghttps://distill.pub/2019/paths-perspective-on-value-learning/ Topics Architecture + Training V-JEPA 2 follows a two-stage recipe. First, an encoder and predictor are pre-trained through self-supervised learning on large-scale video. Instead of attempting to generate missing or future pixels directly, the model predicts representations of unseen or future portions of a video in a learned latent space. This encourages the system to focus on higher-level, predictable structure—such as objects, motion, geometry, spatial relationships, and physical interactions—rather than spending capacity modeling every visual detail. The central idea is that a useful representation of the physical world may emerge precisely because the model is forced to answer: What information about the present is necessary to predict the future? At scale, this creates representations that can support not only visual understanding, but also prediction and downstream physical reasoning. Second, the pretrained world model is connected to robotics through action-conditioned training using a comparatively small amount of robot interaction data. This teaches the model something passive video cannot provide directly: how a robot’s actions affect future states of the world. The resulting model can evaluate candidate actions in relation to a desired target state, enabling goal-conditioned planning in previously unseen environments. Meta reports using 62 hours of robot interaction data from the DROID dataset for this stage. The result suggests a possible scaling recipe for robotics: learn broad physical priors from abundant video, then connect those representations to action using much smaller quantities of expensive robot data. This separates two problems that are often learned together: How does the world work?andHow can this robot act within it? From Prediction to Planning A world model becomes especially interesting when its predictions can be used to choose actions. If a model can predict what the world may look like after taking a particular action, a planner can compare different possible futures and choose actions expected to move the system toward a goal. This reframes robot control from directly learning: Observation → Action toward a richer loop: Observation → Prediction → Evaluation → Action V-JEPA 2 provides one example of this transition. We’ll also discuss WorldPlanner, which approaches the prediction-to-action problem more explicitly by combining an action-conditioned visual world model with planning algorithms such as Monte Carlo Tree Search and Model Predictive Control. Together, these approaches raise a broader question: Could prediction become the computational substrate on top of which planning and physical reasoning are built? Prediction, Value + Action Prediction alone does not determine behavior. A model may be able to imagine many possible futures, but an intelligent agent must still determine which future it wants. This motivates a decomposition of physical intelligence into: Prediction — What will happen?Value — Which outcomes are desirable?Action — What should I do? Daniel will discuss how predictive world models interact with value learning, planning, and control—and whether these capabilities should remain separate components or eventually be learned jointly within a single system. This also connects modern world-model research with older ideas from reinforcement learning and model-based control: search through possible futures, evaluate their value, and select actions accordingly. Physical AI Post-Training Pre-training may teach a model broad representations of the physical world. But capable physical agents will still need to learn how to perform tasks, improve from feedback, adapt to new environments, and acquire new behaviors after pre-training. That makes post-training a central problem for Physical AI. Frontier AI systems increasingly rely on reinforcement learning, environment interaction, synthetic experience, task-specific feedback, and verifiable outcomes after large-scale pre-training. Robotics faces a similar challenge—but physical interaction is dramatically slower, more expensive, and harder to scale. This makes simulation difficult to avoid. At the same time, simulated environments inevitably approximate reality. So what role should simulation play? Can reinforcement learning in simulation produce robust physical intelligence? How much simulated experience transfers to real robots? Can learned world models supplement traditional physics simulators? Could predictive models themselves eventually generate useful imagined experience for robot learning? And how should simulation, demonstrations, real-world interaction, and world-model prediction fit together in a scalable Physical AI training stack? Why It Matters Today’s robotic learning systems often depend on large collections of demonstrations or interactions gathered for particular tasks, environments, and embodiments. Predictive world models suggest a different path. If models can acquire broad physical understanding from passive video, robot interaction data may increasingly become a way of teaching a model how to act, rather than forcing it to learn the structure of the physical world from scratch. V-JEPA 2 is an important example of this idea. But the broader question goes beyond any one architecture: Can prediction become a foundation for physical intelligence? If scalable predictive models can learn useful representations of dynamics, transfer those representations across embodiments, imagine the consequences of actions, and support planning, they may become a central component of future general-purpose robot systems. That raises an even larger possibility: a Physical AI stack in which large-scale observational pre-training provides the model’s understanding of the world, while planning, value learning, reinforcement learning, simulation, and relatively lightweight robot post-training teach it how to operate within that world. Open Discussion + Q&A