Find What You Mean: Zero-Shot Visual Grounding with Qwen3-VL and Transformers

TL;DR

Qwen3-VL can turn natural-language spatial prompts into structured bounding boxes without training a detector.
The current transformers chat-template API handles image inputs directly, so you no longer need the older Qwen2.5-specific preprocessing helper.
This tutorial uses Qwen/Qwen3-VL-8B-Instruct as the default model, with smaller and larger Qwen3-VL variants noted when hardware changes.

Abstract

Object detection usually starts with a fixed label set: cars, people, dogs, traffic lights. Visual grounding starts from language instead. You ask for “the cyclist closest to the bus” or “the cupcake with chocolate chips,” and the model returns the region that best matches the phrase. This post shows how to use Qwen3-VL and the latest Hugging Face transformers API for zero-shot object detection, precise grounding, and relationship-aware grounding, then draw the model’s boxes on the source image.

By the end, you will have a compact inference utility you can reuse for quick spatial-understanding experiments before deciding whether you need a trained detector.

Requirements

Python 3.10+
torch
transformers>=5.12.1
accelerate
pillow
requests
matplotlib

pip install -U "transformers>=5.12.1" accelerate pillow requests matplotlib

For GPU inference, install the torch build that matches your CUDA environment from the PyTorch installation guide.

Prerequisites

You know how to run a Hugging Face model with from_pretrained.
You are comfortable with GPU memory tradeoffs for multimodal models.
You understand that zero-shot boxes are useful for exploration, not a replacement for validated detection metrics.

Why Qwen3-VL Now

The original draft used Qwen/Qwen2.5-VL-3B-Instruct and the older Qwen2_5_VLForConditionalGeneration class. That works for Qwen2.5, but it is no longer the most current Qwen vision-language path.

For this rewrite, the model choice is:

MODEL_ID = "Qwen/Qwen3-VL-8B-Instruct"

That is the best default for this tutorial because it is:

an official Qwen3-VL instruct checkpoint
Apache-2.0 licensed
supported by the current transformers Qwen3-VL class
large enough for useful grounding behavior without jumping to the 30B or 235B variants

If you have less memory, try Qwen/Qwen3-VL-4B-Instruct or Qwen/Qwen3-VL-2B-Instruct. If you have much more memory and want stronger reasoning, try Qwen/Qwen3-VL-30B-A3B-Instruct.

Approach

The workflow is simple:

Load Qwen3-VL and its processor.
Send an image plus a spatial prompt through the chat template.
Ask the model to return only JSON.
Parse the boxes.
Draw the boxes on the original image.

This keeps the tutorial focused on the mechanism: using a general vision-language model as a zero-shot spatial interface.

1. Load Qwen3-VL

import torch
from transformers import AutoProcessor, Qwen3VLForConditionalGeneration

MODEL_ID = "Qwen/Qwen3-VL-8B-Instruct"

model = Qwen3VLForConditionalGeneration.from_pretrained(
    MODEL_ID,
    dtype="auto",
    device_map="auto",
)

processor = AutoProcessor.from_pretrained(MODEL_ID)

For larger image batches or longer prompts, use FlashAttention 2 if your environment supports it:

model = Qwen3VLForConditionalGeneration.from_pretrained(
    MODEL_ID,
    dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
    device_map="auto",
)

2. Create a Grounding Prompt

Vision-language models are sensitive to output instructions. For bounding boxes, ask for a small JSON schema and nothing else.

def make_grounding_prompt(task: str, image_width: int, image_height: int) -> str:
    """Create a prompt that asks for grounded boxes in source-image pixels."""
    return f"""
You are doing visual grounding.

Image size: width={image_width}, height={image_height}.

Task: {task}

Return only a JSON array. Each item must have:
- "label": a short string
- "bbox_2d": [x1, y1, x2, y2] in pixel coordinates for the original image
- "confidence": a number from 0 to 1

Rules:
- Do not include Markdown fences.
- Do not explain your reasoning.
- If nothing matches, return [].
""".strip()

The “return only JSON” instruction matters. It makes the response easier to parse and keeps the visualization code boring, which is exactly what you want.

3. Run Inference

Qwen3-VL’s current processor can apply the chat template and prepare image inputs in one call.

from PIL import Image


def ground_image(
    image: Image.Image,
    task: str,
    max_new_tokens: int = 512,
) -> str:
    """Run a visual-grounding prompt and return the model's raw text."""
    image = image.convert("RGB")
    width, height = image.size
    prompt = make_grounding_prompt(task, width, height)

    messages = [
        {
            "role": "user",
            "content": [
                {"type": "image", "image": image},
                {"type": "text", "text": prompt},
            ],
        }
    ]

    inputs = processor.apply_chat_template(
        messages,
        tokenize=True,
        add_generation_prompt=True,
        return_dict=True,
        return_tensors="pt",
    ).to(model.device)

    generated_ids = model.generate(
        **inputs,
        max_new_tokens=max_new_tokens,
        do_sample=False,
    )

    generated_ids = [
        output_ids[len(input_ids) :]
        for input_ids, output_ids in zip(inputs.input_ids, generated_ids)
    ]

    return processor.batch_decode(
        generated_ids,
        skip_special_tokens=True,
        clean_up_tokenization_spaces=False,
    )[0]

This uses greedy decoding (do_sample=False) because structured output is easier to debug when the generation is deterministic.

4. Parse and Draw Boxes

Models sometimes return a dictionary with an objects key instead of a bare list. The parser below accepts both forms, clamps coordinates to the image, and ignores malformed rows.

import json
import re
from dataclasses import dataclass

import matplotlib.patches as patches
import matplotlib.pyplot as plt
from PIL import Image


@dataclass(frozen=True)
class GroundedBox:
    label: str
    bbox_2d: tuple[float, float, float, float]
    confidence: float | None = None


def extract_json(text: str) -> object:
    """Extract the first JSON object or array from a model response."""
    fenced = re.search(r"```(?:json)?\s*(.*?)```", text, flags=re.DOTALL)
    if fenced:
        text = fenced.group(1)

    text = text.strip()
    if text.startswith("[") or text.startswith("{"):
        return json.loads(text)

    match = re.search(r"(\[.*\]|\{.*\})", text, flags=re.DOTALL)
    if not match:
        raise ValueError(f"No JSON found in response: {text[:200]}")

    return json.loads(match.group(1))


def clamp(value: float, lower: float, upper: float) -> float:
    return max(lower, min(value, upper))


def parse_grounded_boxes(text: str, image_size: tuple[int, int]) -> list[GroundedBox]:
    """Parse model JSON into validated boxes."""
    width, height = image_size
    data = extract_json(text)

    if isinstance(data, dict):
        data = data.get("objects", data.get("boxes", []))

    if not isinstance(data, list):
        raise ValueError("Grounding response must be a JSON list or a dict containing a list.")

    boxes: list[GroundedBox] = []
    for item in data:
        if not isinstance(item, dict) or "bbox_2d" not in item:
            continue

        coords = item["bbox_2d"]
        if not isinstance(coords, (list, tuple)) or len(coords) != 4:
            continue

        x1, y1, x2, y2 = [float(value) for value in coords]
        x1 = clamp(x1, 0, width)
        x2 = clamp(x2, 0, width)
        y1 = clamp(y1, 0, height)
        y2 = clamp(y2, 0, height)

        if x2 <= x1 or y2 <= y1:
            continue

        confidence = item.get("confidence")
        if confidence is not None:
            try:
                confidence = float(confidence)
            except (TypeError, ValueError):
                confidence = None

        boxes.append(
            GroundedBox(
                label=str(item.get("label", "object")),
                bbox_2d=(x1, y1, x2, y2),
                confidence=confidence,
            )
        )

    return boxes


def plot_grounded_boxes(image: Image.Image, boxes: list[GroundedBox]) -> None:
    """Display an image with grounded boxes overlaid."""
    fig, ax = plt.subplots(figsize=(10, 8))
    ax.imshow(image)
    ax.axis("off")

    for box in boxes:
        x1, y1, x2, y2 = box.bbox_2d
        label = box.label
        if box.confidence is not None:
            label = f"{label} ({box.confidence:.2f})"

        rect = patches.Rectangle(
            (x1, y1),
            x2 - x1,
            y2 - y1,
            linewidth=2,
            edgecolor="red",
            facecolor="none",
        )
        ax.add_patch(rect)
        ax.text(
            x1,
            max(0, y1 - 6),
            label,
            color="white",
            fontsize=10,
            bbox={"facecolor": "red", "alpha": 0.8, "pad": 2},
        )

    plt.show()

5. Load Images from URLs

The examples below use public images so you can run them without preparing a dataset.

from io import BytesIO

import requests
from PIL import Image


def load_image(url: str) -> Image.Image:
    """Download an image URL as an RGB PIL image."""
    response = requests.get(url, timeout=20)
    response.raise_for_status()
    return Image.open(BytesIO(response.content)).convert("RGB")

Example 1: Zero-Shot Object Detection

Start broad. Ask the model to find road users and vehicles without giving it a fixed detector head.

traffic_image = load_image(
    "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg"
)

raw_response = ground_image(
    traffic_image,
    "Detect visible vehicles, pedestrians, traffic signs, and other important road objects.",
)

print(raw_response)
boxes = parse_grounded_boxes(raw_response, traffic_image.size)
plot_grounded_boxes(traffic_image, boxes)

This is useful for exploratory labeling. You can quickly see whether the model understands the scene vocabulary before investing in a labeled detection dataset.

Example 2: Precise Object Grounding

Now make the prompt narrower. Instead of asking for all objects, ask for a specific phrase.

pedestrian_image = load_image(
    "https://images.unsplash.com/photo-1500530855697-b586d89ba3ee?w=1200"
)

raw_response = ground_image(
    pedestrian_image,
    "Find the person who is most visually prominent in the scene.",
)

print(raw_response)
boxes = parse_grounded_boxes(raw_response, pedestrian_image.size)
plot_grounded_boxes(pedestrian_image, boxes)

This is the core difference between visual grounding and ordinary object detection: the query can be an open-ended phrase, not just a class name.

Example 3: Relationship-Aware Grounding

Grounding gets more interesting when the phrase depends on relationships between objects.

street_image = load_image(
    "https://images.unsplash.com/photo-1494526585095-c41746248156?w=1200"
)

raw_response = ground_image(
    street_image,
    "Find the object closest to the front door.",
)

print(raw_response)
boxes = parse_grounded_boxes(raw_response, street_image.size)
plot_grounded_boxes(street_image, boxes)

Relationship prompts are harder than category prompts. If a model fails here, inspect the raw response before changing code. Often the issue is prompt ambiguity rather than the drawing utility.

Practical Guidance

Use zero-shot grounding when you need a fast read on whether a VLM understands your domain. It is especially helpful for:

bootstrapping annotation candidates
exploring rare classes before training a detector
checking whether a natural-language concept is visually grounded
prototyping UI flows where users ask for regions in an image

Use a trained detector when you need repeatable metrics, high throughput, or strict class definitions. A VLM can suggest regions, but a validated detector is still the right tool for production object counting, safety-critical alerts, and benchmarked detection systems.

Failure Modes and Caveats

The most common failure is invalid JSON. Keep the schema small, use greedy decoding, and parse the raw response separately from visualization.

The second failure is coordinate mismatch. Tell the model the source image size and ask for pixel coordinates. If your outputs look consistently scaled, add a normalization step or ask for [0, 1000] coordinates and convert them yourself.

The third failure is over-grounding. When the phrase is ambiguous, the model may return several plausible boxes. Tighten the query with visible attributes: color, position, relationship, or count.

Finally, do not treat confidence as calibrated probability. It is a model-generated number, not a detector score trained with localization loss.

Sanity Check

Before adapting this to your own images, verify:

transformers.__version__ is at least 5.12.1
Qwen3VLForConditionalGeneration imports successfully
one example returns valid JSON
parse_grounded_boxes() returns at least one GroundedBox
the plotted boxes align with the original image dimensions

Summary

Qwen3-VL makes visual grounding feel more like asking a question than configuring a detector. With the current transformers chat-template API, the code path is short: load the model and processor, pass an image plus a structured prompt, parse JSON, and draw boxes. That gives you a practical bridge between free-form language and image regions, especially during early dataset exploration.

Next Steps

Try the same helper on your own unlabeled images and save the JSON as annotation candidates.
Compare Qwen/Qwen3-VL-4B-Instruct, Qwen/Qwen3-VL-8B-Instruct, and Qwen/Qwen3-VL-30B-A3B-Instruct on the same prompts.
If you need production detection, use the VLM outputs to accelerate labeling, then train a detector such as RT-DETR, D-FINE, or YOLO on reviewed boxes.