SEW-Multimodal-AMR: RGB, Thermal, 4D Radar and ToF on One Robot, Loaded in FiftyOne

One robot, six ways of sensing the same scene, and a label set that says what was in front of it. SEW-EURODRIVE’s autonomous mobile robot carries an RGB camera, a thermal camera, a time-of-flight (ToF) depth sensor, 4D radar, two 2D laser scanners and an ultrasonic array. In this FiftyOne conversion you can open one frame and compare what the thermal camera, the radar and the RGB camera each made of it.

What is in it

SEW Multimodal AMR is the labeled test set of the SEW-EURODRIVE Multimodal AMR dataset, converted to native multimodal MCAP episodes. It has 3,151 labeled frames in 55 episodes, one per source recording session. Episodes run from 6 to 317 frames, and every modality shares one frame clock. The recordings span three seasons, six weather conditions, and day, dawn and night.

Stream	What it holds
/rgb-camera	RGB, 1224x1024
/thermal-camera	Thermal, 640x512
/rgb-camera-calibrated, /thermal-camera-calibrated	Each camera registered to the thermal frame
/tof-depth, /tof-amplitude	Time-of-flight depth and amplitude, 16-bit PNG
/tof-points	Time-of-flight point cloud (foxglove.PointCloud)
/radar-points	Radar point cloud (foxglove.PointCloud)
/radar-azimuth-abs, /radar-azimuth-phase, /radar-doppler-abs, /radar-doppler-phase, /radar-elevation-abs, /radar-elevation-phase	Six radar heatmap views
/laserscan-left-front, /laserscan-right-back	The two laser scanners (foxglove.LaserScan)
/ultrasonic.plot	The four-sensor ultrasonic range array
/annotations-3d	9,813 3D cuboids (foxglove.SceneUpdate)
/rgb-annotations, /thermal-annotations	2D annotations on each camera (foxglove.ImageAnnotations)
/conditions	Season, weather and time of day

The cuboids cover person, bicycle, doll, slidecar, curb and vegetation. Curb and vegetation are annotated in 3D only. Each episode also carries the fields session, season, weather, daytime, num_frames, duration and num_boxes_3d.

Compare modalities on the same frame

Load the dataset and launch the App:

import fiftyone as fo
import fiftyone.utils.huggingface as fouh

dataset = fouh.load_from_hub(
    "Voxel51/SEW-Multimodal-AMR",
    name="SEW-Multimodal-AMR",
    persistent=True,
)
fo.launch_app(dataset)

Open an episode and step through it. The calibrated RGB and thermal streams are registered to the same frame, so you can check whether a person or a bicycle that is clear in RGB at dawn is still clear in thermal, and what the radar points and heatmaps show at the same instant.

To see which conditions the episodes cover, and then start with the busiest scenes, query the per-episode fields:

print(dataset.distinct("weather"))

busiest = dataset.sort_by("num_boxes_3d", reverse=True)
fo.launch_app(dataset, view=busiest)

Know the limits before you use it

Most episodes are not continuous video. Outside one dense 910-frame block, the test set samples every tenth source frame, so you get a low-rate timeline. That suits comparing modalities frame by frame. It does not suit anything that needs motion between consecutive frames.

Three conversion choices change what you see:

• Depth and amplitude are 16-bit PNG at native scale, so depth values are millimetres over a 0 to roughly 12,500 range. A viewer that maps the full 16-bit range renders them dark. The preview above is contrast stretched for display only.
• The raw radar cubes, which shipped as MATLAB .mat files, are not included. They are two thirds of the source payload and no viewer renders them. The derived heatmaps and point clouds are.
• Cuboids stay in the source’s left-handed sensor frame, with the h w l and x y z fields unchanged. If your pipeline expects a right-handed convention, convert them yourself.

The source ships no class-name file for the YOLO labels. The conversion maps the four YOLO indices to person, bicycle, slidecar and doll by matching per-class box counts against the KITTI labels, which name their classes directly.

The source dataset is released by SEW-EURODRIVE GmbH & Co. KG under CC-BY-SA-4.0, and this conversion uses the same license. For another multimodal robot dataset in FiftyOne, with synced camera and tactile streams, see DreamTac, a Franka arm dataset with gel fingertip video. For more of my FiftyOne dataset notes, see my Voxel51 author page, and to curate your own robot data in the same tool, try Voxel51.

Frequently asked questions

What is in the SEW Multimodal AMR dataset?

It is the labeled test set of the SEW-EURODRIVE Multimodal AMR dataset: 3,151 frames split into 55 episodes, one per source recording session. Six sensing modalities ride one autonomous mobile robot: RGB, thermal, time-of-flight, 4D radar, two 2D laser scanners and an ultrasonic array.

What conditions does SEW Multimodal AMR cover?

The episodes span three seasons, six weather conditions, and day, dawn and night. Each episode has a conditions stream that names the season, weather and time of day, plus session, season, weather and daytime fields you can filter on.

How do I load SEW Multimodal AMR in FiftyOne?

Call load_from_hub with Voxel51/SEW-Multimodal-AMR and persistent=True, then launch the App. Each episode is a native multimodal MCAP file with every modality on one frame clock.

Does SEW Multimodal AMR include the raw radar data?

No. The raw radar cubes shipped as MATLAB files are left out because they are two thirds of the source payload and no viewer renders them. The conversion keeps the derived radar point clouds and six heatmap views.

Is SEW Multimodal AMR continuous video?

Mostly not. Outside one dense 910-frame block, the test set samples every tenth source frame, so most episodes are a low-rate timeline rather than continuous video.