Egocentric-EMG-Force: First-Person RGB-D With Forearm EMG, in FiftyOne

This dataset records the muscle signal in a person’s forearms while they work through household chores. The wearer has a depth camera and an EMG band on each wrist, and sweeps a floor, loads a washing machine or applies hand cream while both bands log the muscle signal.

What is in it

The data comes from the Lo6yu/egocentric_dataset daily activity release. The Voxel51 version converts it to native multimodal MCAP episodes. The set has eight episodes and 38.2 minutes of recording, covering eight household tasks: sweeping and mopping, tidying a bedroom, a dining room and a living room, loading a washing machine, washing hands at a sink, wiping hands on paper towel, and applying hand cream.

Each episode is cut into subtasks that a reviewer described in English. There are 122 segments over 121 distinct phases, and they read as instructions: pick up dustpan and broom, fit the trash bag into the basin, restock paper rolls.

Across the whole set that comes to 67,185 camera and depth frames, 2,539,493 EMG samples, 60,254 hand skeletons and 133,721 per-finger force readings.

Stream	What it holds
/camera	First-person view, 640x480, foxglove.CompressedVideo
/depth	Matching Z16 depth frame, 16-bit PNG
/emg-left.plot, /emg-right.plot	Eight channels per wrist at roughly 550 Hz
/wrist-imu-left.plot, /wrist-imu-right.plot	Accelerometer, gyro and orientation from each band
/finger-force-{left,right}	Newtons for thumb, index, middle, ring and pinky
/finger-contact-{left,right}.plot	Per-finger contact confidence
/finger-flexion-{left,right}.plot	Mean joint flexion per finger
/hand-keypoints	21-point skeleton for each visible hand, foxglove.SceneUpdate
/subtask, /subtask.plot	Each segment’s English label at the frame it begins
/instruction	The task the episode is carrying out

Episodes also carry fields you can filter on: task, task_description, scene_description, subtasks, phases, objects, num_emg_samples, num_force_frames, peak_finger_force, duration and fps, among others.

Scrub the first-person view against both arms

Load it and launch the App:

import fiftyone as fo
import fiftyone.utils.huggingface as fouh

dataset = fouh.load_from_hub(
    "Voxel51/Egocentric-EMG-Force",
    name="Egocentric-EMG-Force",
    persistent=True,
)
fo.launch_app(dataset)

Open an episode and play it. The first-person video, the EMG channels from both wrists and the finger force traces share one timeline. Scrub to the frame where a subtask label appears, then read what each forearm was doing as the hand closed on the object.

To start with the firmest contact, filter on the peak finger force:

view = dataset.match({"peak_finger_force": {"$gt": 40}})
fo.launch_app(dataset, view=view)

The phases field is a list of subtask labels, so you can also match on text. This returns episodes that involve folding:

view = dataset.match({"phases": {"$elemMatch": {"$regex": "fold"}}})

Know the limits before you train on it

The force is an estimate. It is derived from hand pose and depth, not read from an instrumented glove, and it saturates at 45 N. A handful of samples across the set reach that ceiling. Treat it as a label from a model of the hand, not as ground truth for pressure.

The set is small. Eight episodes at 38.2 minutes total is enough to inspect the signals and prototype a pipeline, and thin for training a policy or a force estimator on its own.

Three details of the conversion affect how you read the signals:

• The wrist EMG leads the camera by 80 ms. The release records that offset and publishes a corrected clock, which is the one used here.
• Hand joints are published in the camera frame as the release computes them. A frame occasionally reports the same hand more than once, and the first detection of each side is the one published.
• hand-cream-application names its contact columns dynamic_finger_force_N_*, where the other seven write finger_contact_force_N_*. The conversion reads both.

Video is re-encoded to Annex-B H.264 without B-frames, because the source stores it as lossless FFV1 in MKV, which the viewer does not decode. Depth is carried as 16-bit PNG at its published scale.

The source dataset is CC BY-NC 4.0, and the conversion uses the same license, so use is limited to non-commercial purposes. The dataset is on Hugging Face at Voxel51/Egocentric-EMG-Force.

If you work with contact signals, Robots Can See. They Still Can’t Feel covers how tactile data is structured across gel sensors, force vectors and taxel grids, and TaF-Tactile-Force shows a force dataset on one clock in FiftyOne.

For more of my FiftyOne dataset notes, see my Voxel51 author page, and to curate your own multimodal data in the same tool, try Voxel51.

Frequently asked questions

What is in the Egocentric-EMG-Force dataset?

Eight episodes and 38.2 minutes of someone wearing a depth camera and an EMG band on each wrist while doing eight household tasks. Each episode has first-person video, depth, eight EMG channels per wrist, a hand skeleton per frame, and an estimate of how hard each finger presses.

Is the finger force in Egocentric-EMG-Force measured?

No. The contact force is an estimate derived from hand pose and depth, not a reading from an instrumented glove. It saturates at 45 N, and only a handful of samples across the set reach that ceiling.

How do I load Egocentric-EMG-Force in FiftyOne?

Call load_from_hub with Voxel51/Egocentric-EMG-Force and persistent=True, then launch the App. To find the firmest contact, match episodes where peak_finger_force is greater than 40.

Is the EMG signal aligned with the video?

Yes. The wrist EMG leads the camera by 80 ms. The source release records that offset and publishes a corrected clock, and the conversion uses the corrected clock so the muscle signal lines up with the frame it belongs to.

Can I use Egocentric-EMG-Force in a commercial product?

No. The source dataset is released under CC BY-NC 4.0, and the FiftyOne conversion is distributed under the same license. Use is limited to non-commercial purposes.