Foundation Models for Robotics: Vision-Language-Action Policies on Real Hardware

Language models improved by scaling data and parameters against an abundant corpus. Robotics has no such corpus. There is no internet-scale archive of robot actions, and there cannot be one, because every trajectory must be produced by a physical machine acting in real time. That single asymmetry shapes everything about how foundation models for robotics are actually built.

The data problem, quantified

Text is effectively free at scale — trillions of tokens already exist. Robot demonstration data costs an operator, a robot and wall-clock time that cannot be compressed, because a five-second task takes five seconds. A skilled teleoperator produces on the order of a few hundred demonstrations per day on a single platform, and those demonstrations are specific to that robot's kinematics, its gripper and its workspace.

Collaborative datasets pooling demonstrations across many institutions and robot types have reached the order of a million trajectories spanning dozens of embodiments. That is a genuine step change for the field, and it is still many orders of magnitude short of what language pretraining consumes. Every serious architecture in this area is a response to that shortfall.

• Inherit from vision-language pretraining, so the model already understands objects, spatial relations and instructions before it sees a single trajectory — the perception problem is largely solved elsewhere and imported.
• Train across embodiments, so data from one arm improves policies on another, and the dataset that matters is the union rather than any single platform's.
• Simulate what can be simulated, and reserve real-world collection for the contact-rich behaviours that transfer poorly.
• Exploit play and unstructured interaction data, which is cheaper per hour than curated demonstration.

Why naive imitation fails, and what action chunking fixes

Behaviour cloning treats control as supervised learning: map observation to action, fit to demonstrations. Its weakness is well characterised and it is not a matter of insufficient data. Small errors move the robot slightly off the demonstrated distribution, where the policy is less certain, which produces larger errors — a compounding process.

Predicting a chunk of future actions rather than a single next action attacks both terms. The policy commits to a short horizon and executes it open-loop before replanning, which reduces the number of independent decisions across the task and produces temporally consistent motion instead of the jitter that per-step prediction generates when consecutive observations are nearly identical.

Vision-language-action models: what the transfer actually is

The central architectural bet is that a model pretrained on web-scale images and text already encodes most of what a robot policy needs to perceive and understand, and that only the mapping to actions must be learned from scarce robot data. Actions are discretised into tokens and emitted through the same head that produces text, so fine-tuning on trajectories is, mechanically, just more sequence modelling.

The pay-off is semantic generalisation that trajectory data alone could never supply. A model that has read about objects it has never manipulated can act on instructions referring to them, because the grounding came from pretraining rather than from demonstrations. Generalisation to novel objects and novel phrasings is markedly better than for policies trained on robot data alone — and generalisation to novel physical dynamics is not, because that knowledge was never in the pretraining corpus.

The latency problem that shapes the system architecture

A billion-parameter vision-language model runs at a few hertz on embedded hardware. A stable manipulator control loop needs hundreds. These cannot be the same loop, and every deployed system resolves it the same way: a hierarchy.

Action chunking is what makes the top tier viable: predicting a one-second horizon at 5 Hz gives the lower tiers something to execute continuously while the next chunk is computed. Safety-critical reactions — force limits, collision stop — live in the bottom tier exclusively, because they must respond in milliseconds and must not depend on a model whose latency is variable.

What these models cannot yet do

Honest assessment matters here, because the field's demonstrations are more impressive than its deployments. Current systems generalise well over semantics — new objects, new instructions, new arrangements. They generalise poorly over physics: a task requiring different force profiles, different friction or different compliance than anything in training is unreliable, because the training data never covered that dimension densely enough.

Success rates on genuinely novel long-horizon tasks remain well below what industrial deployment requires, and error recovery is weak — these models are trained on successful demonstrations and have limited experience of what to do after something has gone wrong. Nor do the clean scaling laws of language modelling clearly hold; performance improves with data, but the exponent is not yet established and the data is not cheap.

How to use this now

The productive posture is neither dismissal nor wholesale adoption. Foundation models are already strong at the semantic layer — interpreting an instruction, identifying the referenced object, proposing a plausible plan — and that layer can be adopted today, feeding conventional planners and controllers that carry the reliability guarantees. Keep classical methods where determinism is required, and structure the system so the learned component can be upgraded independently as capability improves. Systems designed with that separation will absorb the next three years of progress. Systems that put a learned policy in the safety path will not.