egodex-test-multimodal: 68-Joint Hand and Body Pose With Egocentric Video in FiftyOne

Every episode in EgoDex comes with a 3D pose for 68 joints across the upper body and both hands, recorded by an Apple Vision Pro while a person does a tabletop task. The pose is not estimated from the video afterward. ARKit produces it on the device at recording time.

What is in it

EgoDex was published at ICLR 2026 by researchers at Apple (source code, paper). This version is the test set, converted to MCAP for the FiftyOne App. It has 3,243 episodes across 111 tabletop manipulation tasks, which comes to 7 hours. Each episode is one .mcap file and one FiftyOne sample.

Every episode streams six channels on a shared timeline:

Channel	What it holds
/camera/image_raw	H.264 egocentric video, 1920x1080, 30 Hz
/camera/calibration	Pinhole intrinsics, logged once. The card gives fx=fy=736.6339, cx=960, cy=540 and no distortion
/skeleton	3D animated skeleton as bone segments in the ARKit world frame, colored by finger
/skeleton_2d	The same skeleton projected onto the video frame
/tf	Camera extrinsics at 30 Hz
/metadata	Task name, task type, language description, objects, verbs, environment string, session name

The raw 4x4 pose for each of the 68 joints (20 upper body, 24 left finger, 24 right finger) stays in the source HDF5 files. They are not stored in this dataset, but hdf5_path on each sample points to the file if your pipeline needs the raw data.

The sample fields are what you filter on in the App:

Field	What it holds
task	Task name, such as basic_pick_place
task_type	reset, reset-free or reversible
language	GPT-4 description of the episode. For reversible tasks, the description that applies to the episode’s direction
language2	The inverse-direction description, empty for non-reversible episodes
objects, verbs	Lists identified by GPT-4, such as ["stapler"] and ["pick", "place"]
environment	Table material, position, background and hand used, in one string
duration_s	Episode length, from about 1.7 s to over 30 s

Watch the skeleton move against the video

Load the dataset and launch the App. The dataset is on Hugging Face as Voxel51/egodex-test-multimodal.

import fiftyone as fo
from fiftyone import ViewField as F
from fiftyone.utils.huggingface import load_from_hub

dataset = load_from_hub("Voxel51/egodex-test-multimodal")
session = fo.launch_app(dataset)

Open an episode. The 3D panel renders the animated skeleton and the Image panel draws the 2D skeleton on the video, both on the 30 Hz clock. Scrub to the frame where a finger closes on an object and check the pose against what the camera sees.

To see how the reversible tasks are described in both directions, filter on the task type and look at the two language fields:

view = dataset.match(F("task_type") == "reversible")
print(view.count(), view.first().language.label, "|", view.first().language2)
fo.launch_app(dataset, view=view)

If you want to see how far a task’s episodes vary in length, sort by duration_s in the same way. There is also a demo Space that runs the dataset in a hosted App.

Know the limits before you use it

This is 7 hours of data. The training set is 725 hours and lives in the source repository, so this release is for exploring, evaluating and debugging a pipeline, not for training from scratch. The episodes are the held-out 1% test partition of the paper’s 99/1 split, and the dataset has no train or validation splits of its own.

The scenes are tabletop manipulation in controlled studio environments. The card lists use cases that need scene diversity or outdoor settings as out of scope.

Three details affect how you use the pose and time:

• The ARKit world frame is not consistent across episodes, because each session sets its own origin. The card says to convert poses to the camera frame with inv(cam_ext) @ joint_tf before computing actions.
• The HDF5 files have no per-frame timestamps. The conversion synthesizes them as session_start_ns + frame_index x (1e9/30), with the start parsed from the session name. About 29 of the 3,243 session names were malformed and needed a regex to recover.
• Language descriptions come from GPT-4 working from collector notes, not from a person describing each episode.

The license is CC-BY-NC-ND 4.0, so commercial use is out. For a look at how another recent multimodal robot dataset handles first-person data, see Egocentric-EMG-Force: First-Person RGB-D With Forearm EMG, in FiftyOne.

For more of my FiftyOne dataset notes, see my Voxel51 author page, and to explore your own multimodal data in the same tool, try Voxel51.

Frequently asked questions

What is in the EgoDex test set?

3,243 episodes across 111 tabletop manipulation tasks, about 7 hours in total. Each episode pairs 1920x1080 egocentric video at 30 Hz with a 3D skeleton, a 2D skeleton overlay, camera extrinsics and a GPT-4 language description.

How is the pose in EgoDex produced?

ARKit on the Apple Vision Pro produces the 3D pose automatically at recording time, with no manual annotation. It covers 68 body and finger joints. The language descriptions are written in two stages: collectors write brief notes, then GPT-4 turns them into one description per episode.

How do I load EgoDex in FiftyOne?

Call load_from_hub with Voxel51/egodex-test-multimodal and then launch the App. Each sample is one episode stored as a single MCAP file. The 3D panel plays the skeleton and the Image panel shows the 2D skeleton on the video.

Can I train a model from scratch on this dataset?

No. This is only the test set, which is 3,243 episodes and 7 hours. Training from scratch needs the full training set, which is 725 hours and available from the source repository.

Can I use EgoDex commercially?

No. The dataset is licensed CC-BY-NC-ND 4.0, and the card lists commercial use as out of scope.