$ teds read --post dreamtac-fiftyone-analysis
DreamTac: 703 Franka Trajectories With Gel Fingertip Video for Robot Learning
DreamTac records a Franka arm with a wrist camera, a third-person camera and two gel fingertips. Watch contact in four synced views in FiftyOne.
You can see the exact moment a robot touches something when its fingertip is a gel pad printed with a marker grid. The grid deforms where the object presses, so contact shows up as a change in the image. DreamTac puts two of these fingertips on a Franka arm, alongside a wrist camera and a third-person camera.
What DreamTac records
DreamTac is a Peking University release of a Franka Emika Panda working through contact-rich tabletop tasks. The Voxel51 conversion turns it into native multimodal MCAP episodes. It holds 703 trajectories across 12 tasks, 506,078 frames and 7.03 hours. Each frame carries all four views, so the set contains 2,024,312 camera frames.
The two fingertips are Xense Photon vision-based tactile sensors. All four cameras record on one 20 fps clock, which is what lets you line up a touch in the fingertip with what the arm was doing.
Each episode has these streams and fields:
| Stream | What it holds |
|---|---|
/camera |
Third-person view at 224x224, as foxglove.CompressedVideo |
/wrist-camera |
The same, from the wrist |
/tactile-left, /tactile-right |
The two Xense Photon fingertips |
/end-effector-pose |
Position and orientation in the Franka base frame |
/end-effector.plot |
The same pose as x, y, z and a rotation vector in radians |
/gripper.plot |
Gripper opening |
/instruction |
Task text, written at each change, so sub-steps appear on the timeline |
Episode-level fields are task, episode_index, instruction, num_frames, duration, fps, end_effector_path_m, end_effector_z_min_m, end_effector_z_max_m, gripper_min, gripper_max and tactile_sensor.
The 12 tasks
The tasks range from 2 episodes to 104. clean_whiteboard has the most episodes at 104 (72.4 minutes, mean path 1.11 m). pick_bread has the most footage: 99 episodes, 105,603 frames and 88.0 minutes, with the longest mean path at 1.83 m. At the other end, cut_banana is 38 episodes and 8.8 minutes, with a mean path of 0.20 m, and insert_usb is 2 episodes and 0.5 minutes.
The instructions are plain sentences, such as “Clean the whiteboard with an eraser.” and “Cut the banana with a knife.” The Mahjong tasks all use the red dragon tile: pick_hongzhong picks it up, and the three push_hongzhong variants push it to a target.
Load it and query it
import fiftyone as fo
import fiftyone.utils.huggingface as fouh
dataset = fouh.load_from_hub(
"Voxel51/DreamTac",
name="DreamTac",
persistent=True,
)
fo.launch_app(dataset)Open an episode and play it. The wrist camera, the third-person camera and both fingertips sit on one timeline, so you can scrub to the frame where the marker grid first distorts and read off what the scene looked like at that instant.
To start with the trajectories that moved the furthest, sort on the path length of the end effector:
view = dataset.sort_by("end_effector_path_m", reverse=True)
fo.launch_app(dataset, view=view)To isolate one task, match on the task field. This pulls up the knife episodes:
from fiftyone import ViewField as F
view = dataset.match(F("task") == "cut_banana")
print(len(view))
fo.launch_app(dataset, view=view)Known limits
The task mix is uneven. The three push_hongzhong tasks are the same push from three different start positions, and insert_usb holds two trajectories against pick_usb’s 102. A model trained on everything will see the large tasks far more often than the small ones. Stratify by task if you want a fair read on insert_usb, pick_hongzhong (18 episodes) or cut_banana.
Arm joint angles are not in the source and are not carried over. If your policy needs joint state, this dataset cannot supply it. The gripper is a single opening scalar.
The conversion drops some trajectories. The release excludes 21 trajectories from a no_tactile batch and 4 that failed source validation, which leaves the 703 you load. Episode and frame counts match the release’s own source_qc_summary.json on all 12 tasks.
The source ships one Zarr v2 group per task in the Open-X-Tactile layout, with every trajectory laid end to end. The per-frame clock restarts at each boundary, so the conversion cuts episodes there and each one starts at zero. The four image streams arrive as raw uint8 arrays and are re-encoded to H.264.
A tactile image shows deformation, and deformation is not force. For a comparison, TaF-Tactile-Force pairs a tactile camera with a six-axis force sensor on one clock. For how gel sensors, force vectors and taxel grids differ as data, see Robots Can See. They Still Can’t Feel.
License and source
The source release is distributed under CC BY 4.0, and the conversion uses the same license. The paper is Dream-Tac: A Unified Tactile World Action Model for Contact-Rich Robot Manipulation (Lou, Ye, Fu, Cen, Chi and Lyu, arXiv:2606.08737). The Voxel51 changes are the MCAP conversion, H.264 encoding of the image streams, splitting the concatenated task groups at their episode boundaries, and encoding pose, gripper and instruction as message streams.
For more of my FiftyOne dataset notes, see my Voxel51 author page, and to curate your own robot data in the same tool, try Voxel51.
Frequently asked questions
What is the DreamTac dataset?
DreamTac is a Peking University release of a Franka Emika Panda doing contact-rich tabletop tasks. It has 703 trajectories across 12 tasks, 506,078 frames and 7.03 hours, with four cameras on one 20 fps clock.
How does DreamTac show contact without a force sensor?
Each fingertip is a gel pad printed with a marker grid, and the grid deforms where an object presses. The moment of contact is visible in the tactile video rather than inferred from a force reading.
How do I load DreamTac in FiftyOne?
Call load_from_hub from fiftyone.utils.huggingface with Voxel51/DreamTac and persistent=True, then launch the App. Sorting by end_effector_path_m shows the trajectories that moved the furthest.
Does DreamTac include robot arm joint angles?
No. Arm joint angles are not in the source release and the conversion does not carry them. The gripper is a single opening scalar, and the end effector pose is in the Franka base frame.
Is the DreamTac task mix balanced?
No. The three push_hongzhong tasks are the same push from three start positions, and insert_usb holds 2 trajectories against 102 for pick_usb. Stratify by task before you compare results across tasks.