Talk to us
$

$ teds read --post detecting-hidden-outliers-visual-ai-embeddings

Detecting Hidden Outliers in Visual AI Datasets with Embeddings and UMAP

Uncover silent data anomalies before production failure. How to use foundation model embeddings, Local Outlier Factor (LOF), and UMAP projections.

ByHarpreet Sahota, Hacker-in-Residence · Principal Developer Relations Engineer · AI ConsultantXLinkedInPublished
Vision & MultimodalAI EngineeringEvaluation & Safety

Your model’s most dangerous vulnerability is rarely a sub-optimal hyperparameter configuration. It is hidden outliers lurking undetected inside your training and evaluation data.

In production, vision models frequently fail on rare scenarios that were never surfaced during development: unusual sensor artifacts, unrepresented object variants, corrupted metadata, or bizarre edge-case interactions. Traditional statistical checks—like tracking image resolution, file size, or mean pixel values—fail to detect semantic and compositional anomalies.

By leveraging foundation model embeddings, density-based anomaly algorithms like Local Outlier Factor (LOF), and dimensionality reduction with UMAP, you can systematically reveal hidden dataset structure and isolate rogue outliers before training begins.

       EMBEDDING-BASED OUTLIER DETECTION PIPELINE
+-------------------------------------------------------------+
| Raw Image Dataset                                           |
|       |                                                     |
|       v                                                     |
| Foundation Model Embedding (CLIP / DINOv2 / SigLIP)          |
|       |                                                     |
|       +-----------------------------+                       |
|       |                             |                       |
|       v                             v                       |
| Local Outlier Factor (LOF)    UMAP 2D Projection            |
| [Quantitative Anomaly Score]  [Visual Cluster Exploration]  |
|       |                             |                       |
|       +-----------------------------+                       |
|       v                                                     |
| Visual Inspection & Triage in FiftyOne                      |
+-------------------------------------------------------------+

The Three Classes of Visual Outliers

When inspecting anomalies in visual datasets, outliers typically fall into three distinct technical categories:

  1. Quality and Sensor Anomalies: Corrupted frames, extreme lens flare, severe compression blockiness, dropped video packets, or infrared sensor bleed. These represent data pipeline defects that should almost always be scrubbed or quarantined.
  2. Semantic and Content Anomalies: Rare, valid physical scenarios that occur with low frequency: an overturned truck on a highway, an animal running across an industrial floor, or a worker wearing non-standard personal protective equipment. These are valuable operational edge cases that must be preserved and reinforced.
  3. Annotation and Formatting Bugs: An image tagged with a label from an entirely different ontology, bounding boxes scaled incorrectly, or labels assigned to inverted coordinate axes.

Identifying which category an outlier belongs to determines whether you prune the sample, fix its metadata, or curate more examples to support it.

Extracting Dense Semantic Embeddings

Pixel-level distances cannot distinguish between a dark cat and a black leather jacket. Foundation vision models trained via self-supervised learning (DINOv2) or contrastive language-image pretraining (CLIP, SigLIP) solve this by encoding high-level semantic, structural, and stylistic properties into dense feature vectors.

import fiftyone as fo
import fiftyone.brain as fob

# Load dataset
dataset = fo.load_dataset("industrial_inspection_v2")

# Compute embeddings using a foundation vision backbone
fob.compute_similarity(
    dataset,
    model="clip-vit-base32-torch",
    embeddings_field="clip_embeddings",
    brain_key="clip_sim"
)

print("Computed dense embeddings across all dataset samples.")

Density-Based Anomaly Scoring with Local Outlier Factor

A common mistake in outlier detection is calculating global Euclidean or Mahalanobis distance from the dataset centroid. Visual datasets are inherently multimodal: images of outdoor scenes form one dense cluster, while indoor scenes form another with completely different local density.

Local Outlier Factor (LOF) solves this by comparing the local density of an image’s embedding to the density of its $k$-nearest neighbors:

$$\text{LOF}k(p) = \frac{\sum{o \in N_k(p)} \frac{\text{lrd}(o)}{\text{lrd}(p)}}{|N_k(p)|}$$

Where $\text{lrd}(p)$ is the local reachability density of point $p$. An LOF score substantially greater than $1.0$ indicates that the sample resides in a sparse region relative to its neighbors, flagging it as an anomaly.

from sklearn.neighbors import LocalOutlierFactor
import numpy as np

# Extract embeddings array from FiftyOne dataset
embeddings = np.array(dataset.values("clip_embeddings"))

# Fit Local Outlier Factor
lof = LocalOutlierFactor(n_neighbors=20, contamination=0.02)
outlier_labels = lof.fit_predict(embeddings)
anomaly_scores = -lof.negative_outlier_factor_

# Store anomaly scores back onto the dataset
dataset.set_values("outlier_score", anomaly_scores.tolist())

# Query the top 1% most severe anomalies
outliers_view = dataset.sort_by("outlier_score", reverse=True).limit(50)
print(f"Isolated top {len(outliers_view)} outlier candidates for inspection.")

Visualizing Dataset Manifolds with UMAP

Quantitative anomaly scores tell you which images are unusual, but dimensionality reduction tells you why.

Uniform Manifold Approximation and Projection (UMAP) preserves both local and global neighborhood structures when projecting high-dimensional embeddings down to a 2D plane:

# Compute 2D UMAP visualization coordinates
fob.compute_visualization(
    dataset,
    embeddings="clip_embeddings",
    method="umap",
    brain_key="umap_2d"
)

# Launch FiftyOne interactive app to inspect clusters
session = fo.launch_app(dataset)

In the interactive 2D scatter plot:

  • Tight, dense clusters represent standard, well-represented operational scenarios.
  • Isolated satellites and rogue points floating outside major clusters pinpoint rare outliers, mislabeled samples, and corrupted images.
Outlier Type UMAP Geometry Typical Root Cause Action Plan
Pipeline Defect Far off the manifold, isolated single point Corrupted sensor frame, black frame Quarantine and remove from training set
Annotation Error Sits inside Cluster A, but labeled as Class B Human labeler typo or ambiguous definition Correct ground truth annotation
Operational Edge Case Small cluster of 3–5 points detached from main body Rare physical scenario in the real world Preserve, balance, and collect companion cases

Turning Anomaly Detection into Practice

Running outlier detection during data ingestion prevents corrupted frames from skewing model weights and ensures that critical real-world edge cases receive explicit attention.

To explore related evaluation workflows, review our guide on curating golden trajectories for red-teaming and analyzing vision datasets.

To get hands-on experience running embedding searches, LOF audits, and UMAP clustering on real datasets, audit Harpreet Sahota’s Coursera course: Hands-on Data Centric Visual AI.

Frequently asked questions

What makes visual outliers dangerous in machine learning datasets?

Visual outliers can corrupt model decision boundaries, cause catastrophic gradients during training, or represent critical real-world edge cases that models fail to recognize in production. Detecting them early prevents both training instability and operational blind spots.

Why are foundation model embeddings better than pixel metrics for outlier detection?

Raw pixel comparisons only capture low-level color and brightness shifts. Foundation model embeddings (such as CLIP, DINOv2, or SigLIP) map images into a structured semantic space where distance reflects conceptual, compositional, and contextual similarity.

How does Local Outlier Factor (LOF) differ from simple Euclidean distance?

Global Euclidean distance assumes uniform density across the dataset. Local Outlier Factor compares the local density of an object to the local densities of its k-nearest neighbors, correctly identifying anomalies even in regions with variable cluster densities.

Where can developers practice embedding-based curation workflows?

Complete hands-on code examples and guided dataset curation labs are available in Harpreet Sahota's Coursera course, Hands-on Data Centric Visual AI.