What is a physical AI data engine?

The pipeline that turns people doing real work into demonstrations a robot can learn from, and turns the deployed robot's own hours back into better demonstrations: capture rigs and consent, stream synchronization, an annotation schema for robot learning, QA gates, co-training on open VLA checkpoints, simulation, and the logging path from the cell back into the corpus.

What kinds of data does a robot policy need?

Five: egocentric human video, teleoperated robot demonstrations, the deployed policy's own autonomous rollouts, synthetic or simulated episodes, and static perception labels. The guide's rule is that simulation supplies volume and coverage while humans supply truth, especially for contact-rich tasks.

How does the annotation schema differ from ordinary image labelling?

Detection labelling asks what is in a frame; robot learning asks what the person was trying to do and whether it worked. The minimum schema carries task and subtask segments from a closed verb vocabulary, object identity, hand and object tracks, contact events, an outcome with reason codes, and take quality flags.

Which open models does the guide recommend starting from?

NVIDIA's Isaac GR00T N1.6 (open, with N1.7 in commercial early access) and Physical Intelligence's pi0 through Hugging Face LeRobot, co-trained with the client's egocentric corpus through an embodiment adapter and a few hundred teleoperated episodes in the robot's own action space.

How is the budget for a first robot cell typically split?

In the guide's illustrative 16-week model for a pick-and-kit cell, capture plus annotation is roughly half the budget, integration about a quarter, and modelling the remainder. Teams that estimate modelling as the majority tend to reach week twelve with a policy and no usable data.

← All whitepapers

Physical AI

The Physical AI Data Engine

Welzin Team · August 28, 2026 · 10 min read

A language model was trained on the written record of humanity. A robot policy has no such record. There is no internet of hands: no archive of how a person threads a cable, seats a connector, folds a shirt, or notices that a part is warped before picking it up. Every team building physical AI in 2026 is therefore also building the thing that creates that record, and the ones that get to production are the ones that treat it as an engineering system rather than a data-collection chore.

This guide is about that system. We call it the physical AI data engine: the pipeline that turns people doing real work into demonstrations a robot can learn from, and turns the robot's own hours back into better demonstrations. It is written for a plant or logistics operator considering a first robot cell, and for a data team that has been asked to supply the training data for one. It assumes no robotics background, and it assumes you would rather copy an architecture than admire one.

What this guide covers

Why the data is the product, with the 2026 evidence
The five data types a policy needs, and which of them simulation cannot supply
Capture: rigs, consent, and the economics of a headset on a worker
Synchronization: the unglamorous step that decides whether the data is usable
The annotation schema for robot learning, and how it differs from detection labels
QA gates: gold sets, agreement, take rejection, and drift
Training: co-training human video with robot demonstrations on open VLA checkpoints
Simulation and synthetic data: where they help and where they lie
Deployment: edge hardware, safety interlocks, and the flywheel
A worked cost model for a first cell
A readiness checklist

1. The data is the product

Three facts from this year settle the argument. Build AI took its egocentric factory dataset from 10,000 hours in November 2025 to roughly a million hours by April 2026, the fastest dataset scaling the field has recorded. Tesla put more than 1,000 Optimus units into its own plants and stated openly that part of the fleet exists to generate training data rather than output. And Georgia Tech's EgoMimic showed that about 90 minutes of first-person human video, co-trained with robot demonstrations, lifted task success by up to 400 percent. The robots are learning from people, and the scarce input is well-recorded, well-labelled footage of people at work.

The consequence for a buyer is that the robot vendor is not the only decision. The vendor supplies an embodiment and a base policy; the performance on your line depends on demonstrations of your tasks, in your lighting, with your parts. That data will be collected either by you, by the vendor at your expense, or not at all.

2. The five data types, and what simulation cannot replace

A production policy draws on five kinds of data. They are not interchangeable.

Egocentric human video. First-person RGB, ideally with depth and hand pose. Cheap, abundant, and the closest thing to a human prior. Cannot be executed by the robot directly; it needs an embodiment bridge.
Teleoperated robot demonstrations. A person drives the actual robot through the task with a leader arm, VR controller, or glove. Expensive per hour, but the data is already in the robot's action space.
Autonomous rollouts. The deployed policy's own attempts, labelled with success or failure. Free once deployed; the flywheel runs on these.
Synthetic and simulated episodes. Generated in a digital twin. Unlimited volume, controllable edge cases, and a persistent gap to reality in contact, friction, deformables, and worn tooling.
Static perception labels. Detection, segmentation, and pose on stills. Needed for the perception stack and for safety monitoring, but a policy does not learn a motion from a bounding box.

The rule of thumb we use: simulation supplies volume and coverage; humans supply truth. Any plan that relies on simulation alone for contact-rich tasks (insertion, cable routing, anything with fabric or foam) will discover the sim-to-real gap on the line.

3. Capture: rigs, consent, and the economics of a headset

The capture rig determines everything downstream. The 2026 baseline is a head-mounted RGB-D camera (Aria-class glasses, or a cap-mounted GoPro plus a depth module), optional wrist cameras for occlusion-free views of the hands, and, for dexterous tasks, a data glove that reports finger pose and contact force. Full-body motion capture is worth it only for humanoid targets where locomotion and posture matter.

Consent is a design input, not a legal afterthought. Recording a worker's shift captures colleagues, screens, and sometimes customers. The programmes that scale (Human Archive's thousand-headset network in India is the reference case) run on explicit per-session consent, on-device blurring of faces and screens, a clear payment per recorded hour, and a retention policy the worker can read. Programmes that skip this stall at the first HR review.

The economics are favourable. A recorded human hour costs a small multiple of the wage; a teleoperated robot hour costs the wage plus the robot, the operator's training, and the cell's downtime. Human video is therefore the bulk of the corpus and teleoperation the calibration set: a few hundred robot episodes per task, not thousands.

4. Synchronization

More projects fail here than in modelling. Every stream (head camera, wrist cameras, depth, glove, and later the robot's joint states) has its own clock, frame rate, and latency. A demonstration where the hand pose lags the video by 60 milliseconds trains a policy that reaches late. The engine needs a single time base, a hardware or visual sync event at the start of each take, per-stream latency calibration, and an automatic check that rejects any take where the residual offset exceeds a frame.

Store the raw streams immutably with their calibration, and treat every derived artifact (resampled frames, aligned poses, task segments) as regenerable. Teams that overwrite raw data with "cleaned" data lose the ability to fix a sync bug retroactively, and there is always a sync bug.

5. The annotation schema

Detection labelling asks "what is in this frame". Robot learning asks "what was the person trying to do, and did it work". The schema has to carry the second question. The minimum we ship with:

Task and subtask segments with start and end timestamps, using a fixed vocabulary agreed with the line (reach, grasp, insert, verify, release).
Object and target identity per segment, linked to the part number where one exists.
Hand and object tracks in 3D where depth allows, in 2D otherwise.
Contact events: first touch, stable grasp, release, and any slip.
Outcome per task: success, recoverable failure, unrecoverable failure, with a reason code.
Take quality flags: occlusion, camera slip, lighting change, off-script behaviour.

Two schema decisions save the most money later. Keep the verb vocabulary small and closed; free-text segment names are unusable for training. And record failures with the same care as successes; a policy that has never seen a failed insertion cannot learn to retry one.

6. QA gates

Labelling quality is measured, not assumed. Every batch runs through the same four gates our annotation practice uses for any dataset: a gold set of pre-labelled takes seeded into the batch to score each annotator; inter-annotator agreement on a sampled overlap, with a threshold per label type (segment boundaries tolerate a few frames, outcome labels tolerate nothing); automatic take rejection on the sync and quality flags above; and drift review, where a senior annotator re-checks a sample every week against the schema, because vocabularies drift within a month of starting.

Because a labeller who never saw the rig cannot judge a take, capture and labelling belong under one roof or under one shared runbook. The split-vendor model, where one company records and another labels, is where most 2025 projects lost a quarter.

7. Training on open checkpoints

The model layer is no longer the hard part. NVIDIA's Isaac GR00T N1.6 is an open vision-language-action model for humanoids with full-body control; N1.7 is in early access with commercial licensing. Physical Intelligence's pi0 is a generalist policy trained across eight robot types and available through Hugging Face LeRobot for fine-tuning. The practical recipe is co-training: a base checkpoint, the client's egocentric human corpus mapped through an embodiment adapter, and a small teleoperated set in the robot's own action space, with the human data weighted to prevent the small robot set from dominating.

Evaluate before deploying, in simulation and on a held-out physical test rig, with the same discipline as any AI system: a fixed scenario set, success rate per subtask, recovery rate after a perturbation, and a regression gate. NVIDIA's Isaac Lab-Arena is the open framework for this. Our companion guide on evaluating AI systems applies unchanged; the metric is a grasp instead of an answer.

8. Simulation and synthetic data

Simulation earns its place in three roles: pre-training locomotion and gross motion where physics is well modelled; generating perturbations (lighting, clutter, part variation) at a volume the real line cannot; and evaluation, where a thousand scripted scenarios expose failure modes before the robot meets a person. Cosmos-class world models extend the second role by generating photoreal variations from a handful of real takes.

It lies in contact. Friction, compliance, cable dynamics, fabric, and worn or dirty tooling are the places where a sim-trained policy looks perfect and then fails on the first real part. Budget real demonstrations for every contact-rich subtask and use simulation for everything around it.

9. Deployment and the flywheel

The cell ships with edge inference on Jetson-class hardware, a hardware safety interlock independent of the model, a monitored perception stack that raises an alert when the scene leaves its training distribution, and a logging path that returns every autonomous attempt, with its outcome, to the data engine. That last item is the flywheel. After the first month the deployed robot is the largest source of demonstrations in the corpus, and the human recordings shift from bulk data to targeted collection for the subtasks where the robot still fails.

10. A worked cost model

Illustrative figures for a single pick-and-kit cell on a mid-size electronics line, over a first 16-week engagement. Treat the ratios as the point, not the absolutes.

Capture: 12 workers, 4 weeks, head and wrist cameras and gloves for two of them; around 1,500 recorded hours.
Annotation: segments, tracks, contacts, and outcomes on roughly 400 curated hours; the rest indexed for retrieval only.
Teleoperation: 300 to 500 robot episodes across the cell's six subtasks.
Simulation: a twin of the cell plus 20,000 perturbed episodes for evaluation.
Training and evaluation: fine-tuning from an open checkpoint on a modest GPU budget; the evaluation rig costs more than the compute.
Integration: PLC and MES handshake, safety review, and operator training.

In that shape, capture plus annotation is roughly half the budget, integration a quarter, and modelling the remainder. Teams that estimate the other way round, with modelling as the majority, are the ones that arrive at week twelve with a policy and no usable data.

11. Readiness checklist

One task, one line metric (cycle time, first-pass yield, or picks per hour) agreed in writing.
A consent and retention policy workers have read, with on-device blurring.
A capture rig with a single time base and an automatic sync check.
A closed verb vocabulary and an outcome label with reason codes.
Gold sets and agreement thresholds defined before the first batch.
An open base checkpoint chosen, with the embodiment adapter tested on a public dataset.
A physical evaluation rig and a fixed scenario set.
A safety interlock independent of the model.
The logging path from the deployed robot back into the corpus.

Welzin builds this engine end to end under its Computer Vision & Data Annotation and Robotics & Physical AI practices, from the capture rig to the fine-tuned policy on the cell, and hands it over with the runbook the client's own team needs to keep the flywheel turning.

Sources

← All whitepapers Talk to a senior engineer →