Hands-on Point Tracking in Video with Meta AI CoTracker3 and FiftyOne

Meta AI released CoTracker3, a state-of-the-art model for tracking points in video. Point tracking represents one of the most critical foundational tasks in computer vision, yet it is often misunderstood or conflated with traditional object tracking.

While object tracking follows bounding boxes or semantic masks around entire items, point tracking follows individual pixels, physical surface features, and keypoint trajectories across consecutive video frames.

Whether you are building physical AI systems, evaluating robotic manipulation policies, compensating for camera motion, or testing physical consistency in generative video world models, point tracking provides the sub-pixel telemetry required to understand motion in three-dimensional space.

Here is a hands-on technical breakdown of how CoTracker3 works and how to convert point tracking inference into an interactive FiftyOne dataset.

       COTRACKER3 POINT TRACKING ARCHITECTURE
+-------------------------------------------------------------+
| Input Video Sequence (Frames 0 to T)                        |
|       |                                                     |
|       v                                                     |
| Dense Feature Pyramid Extraction (CNN / Transformer)        |
|       |                                                     |
|       v                                                     |
| Query Points Selection (Grid or Custom Keypoints)           |
|       |                                                     |
|       v                                                     |
| Sliding-Window Transformer Trajectory Estimation            |
|   - Multi-frame correlation matching                        |
|   - Occlusion and visibility prediction                     |
|       |                                                     |
|       v                                                     |
| Output Trajectory Tensor (T, N, 2) + Visibility (T, N)      |
|       |                                                     |
|       v                                                     |
| FiftyOne Dataset Visualization (Interactive Inspection)     |
+-------------------------------------------------------------+

Point Tracking vs. Object Tracking: The Structural Difference

In standard object detection and multi-object tracking (MOT):

• A model detects a person, car, or tool using a bounding box.
• A tracking algorithm (e.g., ByteTrack, DeepSORT) associates boxes across frames using Kalman filters and appearance embeddings.

This approach fails when you need to know how an object is physically moving or deforming:

• A robotic gripper grasping an elastic object changes shape continuously.
• A dancing human’s limbs articulate rapidly, causing bounding boxes to fluctuate wildly without capturing joint mechanics.
• A vehicle turns, rotating its surface features into and out of line-of-sight.

Point tracking operates at the physical coordinate level. Given an initial set of 2D coordinates $(x_0, y_0)$ at frame $t_0$, the model tracks the exact trajectory $(x_t, y_t)$ across every subsequent frame $t$, even when the physical point is temporarily occluded behind another object.

How CoTracker3 Operates

CoTracker3 formulates point tracking as an iterative trajectory refinement task using an efficient transformer architecture:

1. Multi-Scale Feature Extraction: The video sequence is processed through a convolutional backbone to generate multi-scale visual feature maps.
2. Dense Point Attention: Rather than tracking each point independently (as in classical Lucas-Kanade optical flow), CoTracker3 tracks groups of points jointly. By attending to relationships between neighboring points, the model leverages physical rigidity and spatial context to resolve ambiguous motion.
3. Sliding-Window Temporal Processing: CoTracker3 processes video in overlapping temporal windows, enabling quasi-online inference for streaming video while retaining long-range temporal context to recover points after lengthy occlusions.
4. Visibility and Confidence Heads: For every tracked point at every frame, the network outputs both the 2D coordinate $(x, y)$ and a visibility flag $v \in [0, 1]$, indicating whether the point is directly visible or occluded.

Real-World Applications in Physical AI and Robotics

Point tracking is rapidly becoming a cornerstone of production AI workflows:

• Robotic Teleoperation and Imitation Learning: Tracking key contact points on human demonstrations to map fingertip trajectories directly to robotic manipulators.
• Physical Invariant Validation in World Models: Generative video models frequently produce physically impossible trajectories (e.g. objects warping or drifting unnaturally). Running CoTracker3 across generated video allows automated pipelines to score physical trajectory consistency.
• Dynamic Camera Ego-Motion Estimation: Tracking stationary background points to separate camera pan and tilt from independent foreground object motion.

Hands-On: Running CoTracker3 and Exporting to FiftyOne

To inspect point tracks visually, convert the raw tensor outputs from CoTracker3 into FiftyOne keypoint or polyline annotations.

import torch
import fiftyone as fo

# Load video sample in FiftyOne
dataset = fo.Dataset("cotracker3_demo")
sample = fo.Sample(filepath="/home/harpreet/workspace/yt_shorts/CoTracker3.mp4")
dataset.add_sample(sample)

# Load Meta AI CoTracker3 model from PyTorch Hub
model = torch.hub.load("facebookresearch/co-tracker", "cotracker3_offline")
model = model.eval().cuda()

# Load video frames as tensor (B, T, C, H, W)
# For demo purposes, assume video_tensor is loaded and normalized
# video_tensor = load_video_tensor(sample.filepath)

# Define grid of query points or sample salient keypoints
# grid_size = 20 (tracking 400 points concurrently)
# pred_tracks, pred_visibility = model(video_tensor, grid_size=20)

# Format trajectories as FiftyOne Keypoint annotations per frame
# FiftyOne displays keypoint trajectories interactively across video playback
print("Converted CoTracker3 trajectories into FiftyOne video annotations.")

Once loaded into FiftyOne, you can scrub through the video timeline, filter tracks by visibility confidence, and inspect exactly where trajectories lose tracking or recover from occlusions.

For related reading on video evaluation and robotics data pipelines, review our field notes on multimodal sensor curation and open-source robotics repositories.

To deepen your understanding of dataset curation and visual AI pipelines, explore Harpreet Sahota’s Coursera course: Hands-on Data Centric Visual AI.

Frequently asked questions

What is the difference between point tracking and object tracking?

Object tracking follows bounding boxes or instance masks around entire semantic objects. Point tracking tracks specific sub-pixel 2D coordinate points across frames, maintaining trajectory correspondence even as objects deform, articulate, or undergo severe occlusion.

What technical improvements does CoTracker3 introduce over CoTracker v1 and v2?

CoTracker3 introduces an efficient transformer architecture that tracks points both quasi-online and offline, achieves superior temporal persistence through long occlusions, and dramatically reduces GPU memory overhead while maintaining high trajectory accuracy.

How is point tracking used in physical AI and robotics?

Point tracking provides foundational motion telemetry for robot visual servoing, dynamic deformation tracking, policy verification, camera motion compensation, and physical invariant testing in world models.

Where can developers inspect point trajectories interactively?

Point trajectories can be formatted as keypoints or polylines and visualized frame-by-frame inside the open-source FiftyOne App.