Caption by Consensus: Ranking Image Descriptions with SigLIP2 and Transformers

TL;DR

SigLIP2 is a modern multilingual vision-language encoder, not a text generator.
You can use it for “captioning” when the task is to choose the best caption from candidate descriptions.
With transformers>=5.12.0, AutoProcessor and AutoModel handle the SigLIP2 image-text scoring path cleanly.

Abstract

Older image-captioning tutorials often start with BLIP because BLIP can directly generate text from an image. That is useful, but it is not always the workflow you want. In many product and evaluation settings, you already have candidate captions from humans, templates, search results, or another model, and the real task is to choose which caption best matches the image. This post shows how to do that with SigLIP2 and the current Hugging Face Transformers API. By the end, you will have a compact caption-ranking utility that you can reuse for dataset cleanup, retrieval, labeling, and lightweight visual quality checks.

Requirements

Python 3.10+
torch
transformers>=5.12.0
pillow
requests

pip install -U "transformers>=5.12.0" torch pillow requests

Prerequisites

You know how to run a Python script or notebook cell.
You understand that image-text similarity is different from free-form caption generation.
You have a CPU or GPU environment that can load a small vision-language model.

Problem

The original BLIP-style workflow answers two useful questions:

“What caption can the model generate for this image?”
“What answer can the model generate for this visual question?”

SigLIP2 answers a different question:

“How well does this image match each piece of text?”

That makes SigLIP2 a better fit for ranking, retrieval, zero-shot classification, multilingual matching, and validation. It does not produce new captions token by token. Instead, it embeds images and text into a shared space and returns a match score.

For a modern tutorial, that distinction matters. Calling SigLIP2 an image-captioning model would hide the most important design choice. The useful version is more precise: use SigLIP2 to select the best caption from a candidate set.

Why SigLIP2 Now

SigLIP2 is Google’s newer SigLIP family for multilingual image-text understanding. It was added to Transformers in 2025 and is documented in the current Transformers model docs. The family includes fixed-resolution checkpoints for standard image-text scoring and NaFlex checkpoints that preserve aspect ratio for inputs where distortion hurts, such as documents or images with text.

For this tutorial, use:

google/siglip2-base-patch16-224 as the default. It is small, fast, and good for a first pass.
google/siglip2-so400m-patch14-384 when you want stronger matching quality and can afford a larger model.
google/siglip2-base-patch16-naflex when native aspect ratio matters more than fixed-size throughput.

The code below uses the base checkpoint so the example stays easy to run. You can change MODEL_ID to one of the larger SigLIP2 checkpoints without changing the surrounding workflow.

Approach

The workflow has four stages:

Load an image.
Write several candidate captions.
Score the image against each caption with SigLIP2.
Sort the captions by match score and inspect the winner.

This is useful when captions come from a template system, a retrieval index, a human labeling queue, or a separate generative model. SigLIP2 becomes the judge that checks whether each sentence actually fits the pixels.

Build a Caption Ranker

1. Load the model and processor

from __future__ import annotations

from dataclasses import dataclass
from io import BytesIO

import requests
import torch
from PIL import Image
from transformers import AutoModel, AutoProcessor

MODEL_ID = "google/siglip2-base-patch16-224"
IMAGE_URL = "https://storage.googleapis.com/sfr-vision-language-research/BLIP/demo.jpg"


@dataclass(frozen=True)
class CaptionScore:
    caption: str
    score: float


device = "cuda" if torch.cuda.is_available() else "cpu"
processor = AutoProcessor.from_pretrained(MODEL_ID)
model = AutoModel.from_pretrained(MODEL_ID).to(device).eval()

AutoProcessor is important here. Current Transformers releases apply the SigLIP2 text preprocessing expected by the checkpoint, including lowercasing and the correct padding/truncation behavior. If you bypass the processor and tokenize text by hand, you can silently degrade retrieval quality.

2. Load an image

def load_image(url: str) -> Image.Image:
    """Load a remote image as RGB PIL data."""
    response = requests.get(url, timeout=10)
    response.raise_for_status()
    return Image.open(BytesIO(response.content)).convert("RGB")


image = load_image(IMAGE_URL)

The example image is the familiar BLIP demo image. Keeping the image constant makes it easier to compare the old generative workflow with the newer contrastive one.

3. Score candidate captions

def rank_captions(image: Image.Image, captions: list[str]) -> list[CaptionScore]:
    """Return captions sorted by SigLIP2 image-text match score."""
    inputs = processor(
        text=captions,
        images=image,
        padding="max_length",
        truncation=True,
        max_length=64,
        return_tensors="pt",
    ).to(device)

    with torch.inference_mode():
        outputs = model(**inputs)

    scores = torch.sigmoid(outputs.logits_per_image).squeeze(0)
    ranked = [
        CaptionScore(caption=caption, score=float(score))
        for caption, score in zip(captions, scores, strict=True)
    ]
    return sorted(ranked, key=lambda item: item.score, reverse=True)

SigLIP-style models return image-text logits. Applying torch.sigmoid gives an independent match confidence for each caption. Treat the scores as relative ranking signals, not calibrated probabilities across every possible image and caption.

4. Try a few descriptions

candidate_captions = [
    "a woman sitting on a beach with a dog",
    "a dog running through deep snow",
    "a group of people eating dinner indoors",
    "a person riding a bicycle through a city street",
    "a woman reading a book beside a sleeping cat",
]

for item in rank_captions(image, candidate_captions):
    print(f"{item.score:.3f}  {item.caption}")

The first caption should rank highest for the demo image. The other captions are useful negatives: they are plausible natural-language image descriptions, but they do not match the image.

Use the Pipeline API for Quick Checks

If you want the shortest possible version, use the zero-shot image classification pipeline:

import torch
from transformers import pipeline

classifier = pipeline(
    task="zero-shot-image-classification",
    model="google/siglip2-base-patch16-224",
    device=0 if torch.cuda.is_available() else -1,
)

results = classifier(
    image,
    candidate_labels=[
        "a woman sitting on a beach with a dog",
        "a dog running through deep snow",
        "a group of people eating dinner indoors",
    ],
)

for result in results:
    print(f"{result['score']:.3f}  {result['label']}")

The pipeline is ideal for demos and smoke tests. The AutoModel version gives you more control when you want embeddings, batching, caching, or custom ranking logic.

When This Is Better Than Caption Generation

Use SigLIP2 caption ranking when:

You already have candidate captions.
You need to validate labels in an image dataset.
You want to retrieve images from text queries or captions from images.
You need a fast scoring model that is easier to reason about than a generative decoder.
You care about multilingual image-text matching.

Use a generative vision-language model instead when:

You need the model to write a new caption from scratch.
You want long-form visual reasoning.
You need conversational visual question answering.
You need structured answers that are not already in your candidate set.

One practical pattern is to combine both: generate several candidate captions with a generative model, then use SigLIP2 to rerank them against the image.

Upgrade the Model

Start with the base checkpoint:

MODEL_ID = "google/siglip2-base-patch16-224"

For stronger matching quality, try:

MODEL_ID = "google/siglip2-so400m-patch14-384"

For aspect-ratio-sensitive inputs, try:

MODEL_ID = "google/siglip2-base-patch16-naflex"

Keep the same AutoProcessor and AutoModel pattern. If you use a large checkpoint on limited hardware, load it with device_map="auto" or a supported quantization backend.

Failure Modes & Caveats

SigLIP2 is a matching model. It cannot invent details that are not already present in a candidate caption.

Caption wording matters. Two semantically similar captions can receive different scores if one uses phrasing closer to the model’s training distribution.

Scores are most useful for ranking captions for the same image. Avoid interpreting a score like 0.84 as a universal truth that can be compared across unrelated datasets without calibration.

Fine-grained attributes can still be hard. If you need reliable matching for small objects, subtle colors, OCR-heavy images, or domain-specific terminology, evaluate on your own labeled examples before relying on the ranking.

Practical Guidance

For production-style use, keep the model loaded once and batch multiple captions per image. If you rank the same caption bank against many images, cache the text embeddings and only recompute image features.

Normalize your candidate captions before scoring. Short, literal captions usually work better than clever prose. For example, prefer “a woman sitting on a beach with a dog” over “a serene seaside moment with companionship.”

For dataset cleanup, do not delete low-scoring samples automatically. Use SigLIP2 to prioritize review: high-confidence matches can pass quickly, while low-confidence or ambiguous examples go to a human queue.

Sanity Check

After running the example, verify:

The highest-ranked caption describes the visible image.
Negative captions score lower than the matching caption.
Changing MODEL_ID does not require changing your scoring code.
Your installed Transformers version is current enough for SigLIP2 processor defaults.

Summary

BLIP-style captioning and SigLIP2 caption ranking solve related but different problems. BLIP generates text. SigLIP2 scores image-text agreement. If your application needs to choose, validate, retrieve, or rerank captions, SigLIP2 gives you a current and compact Transformers workflow that is easier to control than open-ended generation.

Next Steps

Swap in your own image and candidate captions.
Try google/siglip2-so400m-patch14-384 on examples where the base model is uncertain.
Pair a generative vision-language model with SigLIP2 reranking when you need both creativity and verification.