Detect What You Can Name: PaliGemma 2 Object Detection with Transformers

TL;DR

PaliGemma 2 can do prompt-driven object detection by generating structured <loc0000> tokens instead of detector head tensors.
The largest current PaliGemma 2 mix checkpoint for this workflow is google/paligemma2-28b-mix-448; use 10b or 3b variants when hardware is tighter.
With current transformers, the core loop is compact: load a processor and model, prompt with detect {label} ; {label}, parse the generated location tokens, and draw boxes.

Abstract

Classic object detectors return boxes and class scores from a dedicated detection head. PaliGemma 2 takes a different route: it treats detection as conditional text generation, so the model emits bounding boxes as location tokens followed by labels. This post shows how to use that behavior with the current Transformers API, turn generated text into pixel boxes, and wrap the result in a reusable function. By the end, you will have a clean baseline for prompt-driven detection that you can extend into a notebook, script, or Gradio demo.

Requirements

Python 3.10+
A Hugging Face account with access accepted for the PaliGemma/Gemma license
A CUDA GPU for the larger checkpoints, or a smaller PaliGemma 2 variant for constrained hardware
transformers>=5.12.0
torch
accelerate
pillow
gradio if you want the demo UI
torchao if you want int4 weight-only loading for the 28B checkpoint

pip install -U "transformers>=5.12.0" torch accelerate pillow gradio torchao
hf auth login

Why PaliGemma 2 Mix 448?

The original notebook used google/paligemma2-3b-mix-224. That is still useful, but the 448 checkpoints give the vision encoder more input resolution, which matters for localization. The current PaliGemma 2 mix family on Hugging Face includes:

Model	Parameters	Why use it
`google/paligemma2-28b-mix-448`	27.7B	Strongest option when you can use quantization or large GPU memory.
`google/paligemma2-10b-mix-448`	9.7B	Good balance when 28B is too large but you still want the higher-resolution `448` variant.
`google/paligemma2-3b-mix-448`	3.0B	The most practical local/dev option.

This tutorial uses google/paligemma2-28b-mix-448 as the main model because it is the largest current PaliGemma 2 mix checkpoint with detection prompt support. If you are experimenting on a smaller GPU, change only MODEL_ID; the rest of the code stays the same.

There is not a single official Hugging Face object-detection leaderboard that directly ranks these PaliGemma 2 mix checkpoints against one another. The choice here is based on current checkpoint availability, input resolution, model size, and explicit detection prompt support in the model cards.

How PaliGemma 2 Represents Detections

PaliGemma detection prompts look like this:

detect person ; bicycle ; dog

The model responds with text shaped like this:

<loc0128><loc0184><loc0660><loc0732> dog ; <loc0201><loc0042><loc0912><loc0408> person

Each detection contains four location tokens in this order:

<loc y_min><loc x_min><loc y_max><loc x_max> label

The location values live on a 0-1024 grid. To draw them on the original image, divide by 1024 and scale y by image height and x by image width.

Load the Model and Processor

For the 28B checkpoint, int4 weight-only loading is the most practical starting point. If you use the 10b or 3b checkpoint and have enough memory, set USE_INT4 = False.

import torch
from transformers import AutoProcessor, PaliGemmaForConditionalGeneration, TorchAoConfig

MODEL_ID = "google/paligemma2-28b-mix-448"
USE_INT4 = True

processor = AutoProcessor.from_pretrained(MODEL_ID)

model_kwargs = {"device_map": "auto"}
if USE_INT4:
    model_kwargs["quantization_config"] = TorchAoConfig("int4_weight_only", group_size=128)
else:
    model_kwargs["torch_dtype"] = torch.bfloat16

model = PaliGemmaForConditionalGeneration.from_pretrained(
    MODEL_ID,
    **model_kwargs,
).eval()

If you want the smaller development model:

MODEL_ID = "google/paligemma2-3b-mix-448"
USE_INT4 = False

Parse Generated Location Tokens

Keep the parser separate from model inference. That makes it easy to unit test and easy to replace later if you add segmentation tokens.

from dataclasses import dataclass
import re


@dataclass(frozen=True)
class Detection:
    label: str
    box_xyxy: tuple[int, int, int, int]


LOC_PATTERN = re.compile(
    r"<loc(?P<y0>\d{4})><loc(?P<x0>\d{4})>"
    r"<loc(?P<y1>\d{4})><loc(?P<x1>\d{4})>"
    r"\s*(?P<label>[^;<>]+?)\s*(?:;|$)"
)


def parse_detections(text: str, image_size: tuple[int, int]) -> list[Detection]:
    """Convert PaliGemma location-token output into pixel-space boxes."""
    width, height = image_size
    detections: list[Detection] = []

    for match in LOC_PATTERN.finditer(text):
        y0 = int(match.group("y0")) / 1024 * height
        x0 = int(match.group("x0")) / 1024 * width
        y1 = int(match.group("y1")) / 1024 * height
        x1 = int(match.group("x1")) / 1024 * width

        detections.append(
            Detection(
                label=match.group("label").strip(),
                box_xyxy=(round(x0), round(y0), round(x1), round(y1)),
            )
        )

    return detections

Run Prompt-Driven Detection

PaliGemma is not a closed-set detector like YOLO or DETR. You give it the object names you care about, and it tries to localize those objects.

from PIL import Image
from transformers.image_utils import load_image


def make_detection_prompt(labels: list[str]) -> str:
    cleaned = [label.strip().lower() for label in labels if label.strip()]
    if not cleaned:
        raise ValueError("Pass at least one label, for example: ['person', 'dog']")
    return "detect " + " ; ".join(cleaned)


def detect_objects(image: Image.Image, labels: list[str], max_new_tokens: int = 256) -> tuple[str, list[Detection]]:
    prompt = make_detection_prompt(labels)
    rgb_image = image.convert("RGB")

    inputs = processor(text=prompt, images=rgb_image, return_tensors="pt").to(model.device)

    with torch.inference_mode():
        generated = model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            do_sample=False,
            cache_implementation="static",
        )

    input_length = inputs["input_ids"].shape[-1]
    generated_tokens = generated[0][input_length:]
    decoded = processor.decode(generated_tokens, skip_special_tokens=False).strip()
    return decoded, parse_detections(decoded, rgb_image.size)


image = load_image(
    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg"
)
raw_output, detections = detect_objects(image, ["car", "wheel", "person"])

print(raw_output)
print(detections)

Notice the skip_special_tokens=False setting. For this task, the <loc...> tokens are the result you need, so do not discard them before parsing.

Draw the Boxes

Keep drawing code boring. It should not know anything about Transformers; it only needs an image and parsed detections.

from PIL import ImageDraw, ImageFont


def clamp(value: int, lower: int, upper: int) -> int:
    return max(lower, min(value, upper))


def draw_detections(image: Image.Image, detections: list[Detection]) -> Image.Image:
    canvas = image.convert("RGB").copy()
    draw = ImageDraw.Draw(canvas)
    font = ImageFont.load_default()
    width, height = canvas.size

    for detection in detections:
        x0, y0, x1, y1 = detection.box_xyxy
        x0 = clamp(x0, 0, width - 1)
        x1 = clamp(x1, 0, width - 1)
        y0 = clamp(y0, 0, height - 1)
        y1 = clamp(y1, 0, height - 1)

        if x1 <= x0 or y1 <= y0:
            continue

        draw.rectangle((x0, y0, x1, y1), outline="red", width=3)
        draw.text((x0 + 4, y0 + 4), detection.label, fill="red", font=font)

    return canvas


annotated = draw_detections(image, detections)
annotated.save("paligemma2-detections.jpg")

Optional Gradio Demo

Once the core functions work, the UI is just a thin wrapper. Let the user pass a comma-separated list of candidate labels.

import gradio as gr


def gradio_detect(image: Image.Image, labels: str):
    label_list = [label.strip() for label in labels.split(",") if label.strip()]
    raw_output, detections = detect_objects(image, label_list)
    annotated = draw_detections(image, detections)
    rows = [
        {"label": detection.label, "box_xyxy": detection.box_xyxy}
        for detection in detections
    ]
    return annotated, raw_output, rows


demo = gr.Interface(
    fn=gradio_detect,
    inputs=[
        gr.Image(type="pil", label="Image"),
        gr.Textbox(value="person, car, dog", label="Objects to detect"),
    ],
    outputs=[
        gr.Image(type="pil", label="Detections"),
        gr.Textbox(label="Raw model output"),
        gr.JSON(label="Parsed boxes"),
    ],
    title="PaliGemma 2 Prompt-Driven Object Detection",
    description="Enter object names, then PaliGemma 2 returns generated location tokens that are parsed into boxes.",
)

demo.launch()

Failure Modes & Caveats

PaliGemma 2 detection is prompt-driven. If you do not name an object, the model is not guaranteed to return it.
The output is generated text, so malformed or partial <loc...> sequences can happen. Keep the raw output visible while debugging.
The Gemma/PaliGemma license applies. Accept the model terms before running the checkpoints.
The 28b model is large. Use int4 loading, a large GPU, or swap to 10b/3b.
Detection quality depends heavily on the label wording. Try singular nouns first, then add more specific phrases if needed.

Practical Guidance

Start with google/paligemma2-3b-mix-448 while building your app, then move up to 10b or 28b.
Use short prompts such as detect car ; person, not long natural-language questions.
Keep do_sample=False for repeatable detections.
Store both the raw generated text and parsed boxes during evaluation. Parser bugs are easier to spot when you can inspect the text.
Treat this as open-vocabulary localization, not a drop-in replacement for a production detector with calibrated confidence scores.

Sanity Check

processor(text=..., images=..., return_tensors="pt") returns input_ids, attention_mask, and pixel_values.
The decoded model output contains <loc0000>-style tokens.
parse_detections(...) returns pixel-space (x0, y0, x1, y1) boxes.
The annotated image draws boxes in the right region of the source image.

Summary

PaliGemma 2 makes object detection feel closer to prompting than configuring a detector head. You name the objects, the model generates location-token text, and a small parser turns that text into boxes. Using the current Transformers API keeps the code compact: AutoProcessor prepares the image and prompt, PaliGemmaForConditionalGeneration generates the detection string, and plain Python handles the rest.

Next Steps

Evaluate prompts across a small labeled image set and track parser failures separately from localization misses.
Add support for segment {object} prompts if masks matter more than boxes.
Fine-tune a pt checkpoint when you need a domain-specific detector instead of a general prompt-driven baseline.