RoboLab-EgoX: Filter 3,368 Labeled Robot Failures in FiftyOne

RoboLab-EgoX keeps the rollouts that went wrong. Of the 4,000 policy rollouts recorded on NVIDIA’s RoboLab manipulation benchmark, 3,368 failed and 632 succeeded. That is 84% failures, and all of them are in the dataset with their success label. If you want to study failure, you do not have to dig for it.

## What is in it

The Voxel51 dataset is the complete DAVIAN-Robotics/RoboLab-EgoX corpus, converted to native multimodal MCAP episodes. The RoboLab paper (arXiv 2604.09860) is titled “A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies”.

There are 4,000 .mcap episodes of 81 frames each at 15 fps, 6.0 hours in total. They cover all 28 RoboLab tasks across all 99 background scenes, with a median of 126 failures per task.

Stream	What it holds
/ego-camera	Wrist view, 640x360 H.264
/exo-left-camera, /exo-right-camera	The two exterior views, 640x360 H.264
/ego-depth, /exo-left-depth, /exo-right-depth	16-bit PNG depth at 320x180
/ego-calibration, /exo-left-calibration, /exo-right-calibration	Intrinsics for each camera
/joint-positions, /actions	Joint positions for 13 joints, and actions, each with a timeline plot
/end-effector-pose	The end-effector pose
/instruction	The task’s language instruction

Each episode carries the fields take_name, task, background, instruction, success, source_episode, num_frames and duration.

Watch the failures

import fiftyone as fo
import fiftyone.utils.huggingface as fouh

dataset = fouh.load_from_hub(
    "Voxel51/RoboLab-EgoX",
    name="RoboLab-EgoX",
    persistent=True,
)

failures = dataset.match({"success": False})
fo.launch_app(dataset, view=failures)

The failures view holds the 3,368 failed takes. Narrow it with task or background to compare how one task fails across scenes, or open the three camera views of a single failure side by side.

Quirks to handle before using depth

No-return depth pixels are 65535, not zero. Mask them before scaling, or the real depth looks black.

Depth is half the camera resolution, 320x180 against 640x360. The published calibration describes the 640x360 colour frame, so halve fx, fy, cx and cy before projecting a depth frame to 3D.

In 153 of the 4,000 takes the source mp4 files hold 80 real frames rather than 81. In those takes all three camera streams are one frame shorter than depth and telemetry, and num_frames still reports 81.

Eight of the 28 tasks have no successful rollout anywhere in the source, so those tasks contribute failures only. Their success rate in this data is zero, which says little about the task itself.

The RoboLab benchmark, tasks and scene assets are released by NVIDIA under Apache 2.0 at NVLabs/RoboLab. DAVIAN-Robotics recorded and published the rollouts, and this subset is also Apache 2.0. The data is at Voxel51/RoboLab-EgoX.

If you plan to rank these takes by a score, read Robot Data Quality Scoring: Why Scores Are Triage, Not Gates first. For more of my FiftyOne dataset notes, see my Voxel51 author page, and to inspect and curate your own multimodal data in the same tool, try Voxel51.

Frequently asked questions

How many RoboLab-EgoX rollouts failed?

3,368 of the 4,000 rollouts failed and 632 succeeded, so 84% are failures. Every rollout is in the dataset with its success label.

What does each take contain?

Three synchronized camera views (a wrist camera and two exterior cameras) with matching 16-bit depth, per-camera intrinsics, joint telemetry, the end-effector pose and the task instruction. Each take is 81 frames at 15 fps.

How do I filter to the failures?

Match on the boolean success field. Setting it to False keeps only the failed rollouts. Each episode also carries task, background and instruction fields to narrow it further.

Why does the depth stream render as flat black in my viewer?

Depth marks no-return pixels with the uint16 ceiling, 65535, rather than zero. That is over 60% of a typical exterior frame and about 12% of a wrist frame, so a viewer that maps the full 16-bit range renders the real depth as black. Mask 65535 before scaling.

Does every task have successful rollouts?

No. Eight of the 28 tasks have no successful rollout anywhere in the source, so those tasks contribute failures only.