Physical AI looks impressive in demos. But nobody can tell you whether a model is ready for production. PhAIL measures that gap – on real robots, with real commercial tasks, using the metrics operations teams actually care about.
UPH – units per hour. How fast the system works.
MTBA – mean time between assists. How long it runs before a human needs to step in.
The first task is bin-to-bin order picking – transferring individual items between containers. Evaluations run on Franka Research 3 arms with Robotiq grippers. More tasks and platforms are coming.
The DROID teleoperation dataset used to fine-tune all models on the leaderboard. 352 episodes, 12GB. Available for non-commercial use.
uv run --with positronic \
positronic-server \
--dataset=@positronic.cfg.ds.phail.phailpip install positronic
positronic-server \
--dataset=@positronic.cfg.ds.phail.phailEvery evaluation run on the leaderboard is a downloadable Positronic dataset – multi-view video and robot telemetry.
uv run --with positronic \
positronic-server \
--dataset=@positronic.cfg.ds.phail.eval_runspip install positronic
positronic-server \
--dataset=@positronic.cfg.ds.phail.eval_runsBrowse individual runs in the Run explorer.
Full evaluation protocol, scoring, and reproducibility details are in the white paper.
PhAIL is governed as an open consortium. Founding partners: Nebius and Toloka.
We're looking for model builders, hardware vendors, and deployers who want to shape how physical AI is measured.