Human Preference Tuning for Small VLMs: SmolVLM2 + DPO in Transformers 5.12.1

TL;DR

Direct Preference Optimization (DPO) lets you align a vision-language model from preferred and rejected answers, without training a separate reward model.
The older HuggingFaceTB/SmolVLM-Instruct example is worth updating to HuggingFaceTB/SmolVLM2-2.2B-Instruct, which is newer, Apache 2.0, and documented with AutoModelForImageTextToText.
For current TRL vision-language DPO, keep the dataset in images, prompt, chosen, and rejected form, pass a processor through processing_class, and set max_length=None so image tokens are not truncated away.

Abstract

Preference tuning is one of the most practical ways to make a vision-language model behave less like a generic captioning engine and more like the assistant you actually want. Instead of asking the model to imitate one answer, DPO asks it to prefer one answer over another for the same image and prompt. This post updates an older SmolVLM DPO notebook into a current Markdown tutorial using SmolVLM2-2.2B-Instruct, transformers==5.12.1, TRL’s VLM-aware DPOTrainer, and LoRA adapters from PEFT.

By the end, you will have a compact preference-tuning script, a small inference check for the saved adapter, and a clearer sense of the failure modes that matter when aligning small multimodal models.

Requirements

Use this setup if you want to run the example as written:

python>=3.10
transformers==5.12.1
a current PyTorch 2.12.x build for your CUDA setup
trl>=0.19.0
peft>=0.13.0
accelerate>=1.0.0
datasets
Pillow
num2words

Install PyTorch first using the selector on pytorch.org, then install the packages used by the examples:

pip install "transformers==5.12.1" "trl>=0.19.0" "peft>=0.13.0" "accelerate>=1.0.0" datasets pillow num2words

If your GPU and CUDA stack support it, install FlashAttention 2 separately and set the attention implementation to flash_attention_2. The script below falls back to the standard attention path so the tutorial is not tied to one CUDA build.

Prerequisites

Basic familiarity with Hugging Face model loading and datasets
A CUDA GPU with enough memory for a 2.2B vision-language model plus LoRA training overhead
Comfort reading preference data with a prompt, preferred answer, and rejected answer
Enough disk space for the model checkpoint, dataset cache, and adapter checkpoints

Problem
Why SmolVLM2
Background
Approach
Example
Inference with the Adapter
Failure Modes & Caveats
Practical Guidance
Summary
Next Steps
Resources

Problem

Supervised fine-tuning teaches a model to imitate a target answer. That is useful, but it is not always the shape of the problem you have.

For alignment work, the question is often comparative:

Given the same image and question, which answer is better?

That is a different signal. A preferred answer might be more truthful, more specific, less speculative, safer, or simply more helpful. DPO gives you a direct way to train on that comparison. Instead of fitting only the chosen answer, it pushes the model to assign higher probability to the chosen response than to the rejected one.

For vision-language models (VLMs), this is especially useful because many bad answers are not obviously wrong from text alone. The model has to connect the image and the instruction, then prefer the answer that respects both.

Why SmolVLM2

The original SmolVLM DPO example used HuggingFaceTB/SmolVLM-Instruct and pinned transformers==4.46.3. That was a reasonable starting point at the time, but the better current baseline is HuggingFaceTB/SmolVLM2-2.2B-Instruct.

SmolVLM2-2.2B-Instruct is a better fit for a modern tutorial because:

It is newer than the first SmolVLM release and supports image, multi-image, video, and text inputs.
It stays small enough to make adapter training approachable.
It uses the current AutoProcessor and AutoModelForImageTextToText path in transformers.
Its model card lists an Apache 2.0 license and documents the required num2words processor dependency.

There are larger VLMs that will score higher on broad benchmarks, but that is not the point here. This tutorial is about a compact model you can actually iterate on while learning preference tuning.

Background

DPO starts with paired preferences. Each training example contains:

an input prompt,
the preferred answer, often called chosen,
the less preferred answer, often called rejected,
and, for a VLM, one or more images.

TRL’s current DPOTrainer supports this directly. For VLMs, the dataset should include either an image column for a single image or an images column for a list of images. The trainer then uses a vision preference collator instead of the plain text collator.

The important detail is sequence length. Text DPO examples are often truncated to a maximum length. With VLMs, careless truncation can remove the image tokens and produce confusing failures. For that reason, the current TRL docs recommend DPOConfig(max_length=None) for VLM training unless you have verified that your limit never cuts through the image portion.

Approach

The workflow is intentionally small:

Load a preference dataset with images, questions, chosen answers, and rejected answers.
Convert each row into TRL’s VLM preference format.
Load SmolVLM2-2.2B-Instruct through the current transformers image-text API.
Add LoRA adapters with PEFT so you are not updating every model weight.
Train with DPOTrainer.
Save the adapter and run one generation check.

The example uses openbmb/RLHF-V-Dataset because it already has the shape we need: image-question preference pairs. For a production run, you would replace this with preference data from your own product or annotation workflow.

Example

The script below is a compact training recipe. It avoids notebook-only display calls, keeps preprocessing in one function, and leaves model loading to DPOTrainer so the trainer can build the right VLM collator around the processor.

from __future__ import annotations

import json
from typing import Any, Final

import torch
from datasets import Dataset, load_dataset
from peft import LoraConfig
from PIL import Image
from transformers import AutoProcessor
from trl import DPOConfig, DPOTrainer


MODEL_ID: Final[str] = "HuggingFaceTB/SmolVLM2-2.2B-Instruct"
DATASET_ID: Final[str] = "openbmb/RLHF-V-Dataset"
OUTPUT_DIR: Final[str] = "smolvlm2-rlhf-v-dpo-lora"
MAX_IMAGE_EDGE: Final[int] = 768
SEED: Final[int] = 42


def supports_bf16() -> bool:
    return torch.cuda.is_available() and torch.cuda.is_bf16_supported()


def attention_implementation() -> str:
    try:
        import flash_attn  # noqa: F401

        return "flash_attention_2"
    except ImportError:
        return "eager"


def resize_image(image: Image.Image) -> Image.Image:
    image = image.convert("RGB")
    image.thumbnail((MAX_IMAGE_EDGE, MAX_IMAGE_EDGE), Image.Resampling.LANCZOS)
    return image


def format_preference_example(example: dict[str, Any]) -> dict[str, Any]:
    payload = json.loads(example["text"])
    question = payload["question"].strip()
    chosen = payload["chosen"].strip()
    rejected = payload["rejected"].strip()

    prompt = [
        {
            "role": "user",
            "content": [
                {"type": "image"},
                {"type": "text", "text": question},
            ],
        }
    ]

    return {
        "images": [resize_image(example["image"])],
        "prompt": prompt,
        "chosen": [{"role": "assistant", "content": [{"type": "text", "text": chosen}]}],
        "rejected": [{"role": "assistant", "content": [{"type": "text", "text": rejected}]}],
    }


def load_preference_splits() -> tuple[Dataset, Dataset]:
    train_dataset, eval_dataset = load_dataset(
        DATASET_ID,
        split=["train[:1000]", "train[-100:]"],
    )

    train_dataset = train_dataset.map(
        format_preference_example,
        remove_columns=train_dataset.column_names,
        desc="Formatting train preferences",
    )
    eval_dataset = eval_dataset.map(
        format_preference_example,
        remove_columns=eval_dataset.column_names,
        desc="Formatting eval preferences",
    )
    return train_dataset, eval_dataset


def main() -> None:
    train_dataset, eval_dataset = load_preference_splits()

    processor = AutoProcessor.from_pretrained(MODEL_ID)
    processor.tokenizer.padding_side = "left"
    if processor.tokenizer.pad_token is None:
        processor.tokenizer.pad_token = processor.tokenizer.eos_token

    peft_config = LoraConfig(
        r=16,
        lora_alpha=32,
        lora_dropout=0.05,
        target_modules=[
            "q_proj",
            "k_proj",
            "v_proj",
            "o_proj",
            "gate_proj",
            "up_proj",
            "down_proj",
        ],
        init_lora_weights="gaussian",
    )

    bf16 = supports_bf16()
    training_args = DPOConfig(
        output_dir=OUTPUT_DIR,
        seed=SEED,
        bf16=bf16,
        fp16=torch.cuda.is_available() and not bf16,
        per_device_train_batch_size=1,
        per_device_eval_batch_size=1,
        gradient_accumulation_steps=8,
        gradient_checkpointing=True,
        learning_rate=1e-5,
        max_steps=200,
        logging_steps=5,
        eval_strategy="steps",
        eval_steps=50,
        save_strategy="steps",
        save_steps=50,
        save_total_limit=2,
        remove_unused_columns=False,
        max_length=None,
        report_to="none",
        model_init_kwargs={
            "dtype": torch.bfloat16 if bf16 else torch.float16,
            "attn_implementation": attention_implementation(),
            "device_map": "auto",
        },
    )

    trainer = DPOTrainer(
        model=MODEL_ID,
        ref_model=None,
        args=training_args,
        train_dataset=train_dataset,
        eval_dataset=eval_dataset,
        peft_config=peft_config,
        processing_class=processor,
    )

    trainer.train()
    trainer.save_model(OUTPUT_DIR)
    processor.save_pretrained(OUTPUT_DIR)


if __name__ == "__main__":
    main()

Two choices in this script are worth calling out.

First, the dataset is not pre-tokenized. Each row stays as images plus structured messages. That gives the processor and TRL’s VLM collator the information they need at batch time.

Second, max_length=None is deliberate. It is tempting to copy a text-only DPO config with max_length=1024, but that can cut away image tokens. Start without truncation, inspect memory use, and only add a limit after you have checked the actual tokenized batches.

Inference with the Adapter

After training, load the base model, attach the saved LoRA adapter, and run generation on one evaluation example. The processor path below uses the same image placeholder style as the training data.

from __future__ import annotations

import json

import torch
from datasets import load_dataset
from peft import PeftModel
from PIL import Image
from transformers import AutoModelForImageTextToText, AutoProcessor


MODEL_ID = "HuggingFaceTB/SmolVLM2-2.2B-Instruct"
ADAPTER_DIR = "smolvlm2-rlhf-v-dpo-lora"


def pick_dtype() -> torch.dtype:
    if torch.cuda.is_available() and torch.cuda.is_bf16_supported():
        return torch.bfloat16
    if torch.cuda.is_available():
        return torch.float16
    return torch.float32


def generate_answer(question: str, image: Image.Image) -> str:
    dtype = pick_dtype()
    processor = AutoProcessor.from_pretrained(ADAPTER_DIR)
    model = AutoModelForImageTextToText.from_pretrained(
        MODEL_ID,
        dtype=dtype,
        device_map="auto",
    )
    model = PeftModel.from_pretrained(model, ADAPTER_DIR)
    model.eval()

    messages = [
        {
            "role": "user",
            "content": [
                {"type": "image"},
                {"type": "text", "text": question},
            ],
        }
    ]

    text = processor.apply_chat_template(
        messages,
        add_generation_prompt=True,
        tokenize=False,
    )
    inputs = processor(
        text=text,
        images=[image.convert("RGB")],
        return_tensors="pt",
    ).to(model.device, dtype=dtype)

    with torch.inference_mode():
        output_ids = model.generate(
            **inputs,
            do_sample=False,
            max_new_tokens=128,
        )

    generated_ids = output_ids[:, inputs["input_ids"].shape[-1] :]
    return processor.batch_decode(
        generated_ids,
        skip_special_tokens=True,
        clean_up_tokenization_spaces=False,
    )[0].strip()


if __name__ == "__main__":
    row = load_dataset("openbmb/RLHF-V-Dataset", split="train[-1]")
    payload = json.loads(row[0]["text"])
    print(generate_answer(payload["question"], row[0]["image"]))

For a real evaluation, do not judge the adapter from one generated answer. Build a small held-out preference set and compare the base model against the adapter with win rate, human review, and task-specific checks.

Failure Modes & Caveats

DPO is compact as code, but it can still give you misleading results.

The first failure mode is weak preference data. If the rejected answer is obviously broken, the model may learn superficial distinctions instead of better visual grounding. Harder pairs are more useful: answers that are both fluent, but differ in factuality, specificity, caution, or instruction following.

The second failure mode is image-token truncation. If you set a text-style max_length, batches may fail or train on damaged inputs. Use max_length=None first, then add limits only after inspecting tokenized examples.

The third failure mode is treating alignment as a single metric. DPO can improve preference behavior while hurting caption detail, refusal behavior, or rare visual reasoning cases. Keep a small suite of qualitative examples alongside any scalar metric.

Finally, remember that SmolVLM2 is small. That is a strength for iteration, but it also means you should not expect it to match the strongest closed or very large open VLMs on difficult reasoning. Use it when the deployment and iteration constraints make a compact model valuable.

Practical Guidance

Start with a small subset and overfit intentionally. If loss does not move on 100-1,000 examples, fix the formatting before scaling the run.

Keep LoRA modest at first. r=16 is a good starting point for the language projection layers. If the adapter underfits after the pipeline is correct, increase rank or train longer before adding more complicated adapter targeting.

Log generated examples throughout training. Preference loss is useful, but alignment work needs examples you can read. Save a fixed set of prompts and compare base, current adapter, chosen, and rejected answers every few checkpoints.

Use your own preference data as soon as possible. Public datasets are useful for learning the mechanics, but the best preference signal comes from the tasks, style, and failure cases your application actually sees.

Summary

Modern VLM preference tuning is much cleaner than the older notebook flow suggests. With SmolVLM2-2.2B-Instruct, AutoProcessor, AutoModelForImageTextToText, PEFT LoRA adapters, and TRL’s VLM-aware DPOTrainer, the core recipe is just: format image preference pairs, preserve the multimodal structure, avoid image-token truncation, and train the model to prefer the better answer.

The important part is not the length of the script. It is the quality of the comparisons. DPO gives you a direct training signal, but the model can only learn the preferences your dataset actually expresses.

Next Steps

Replace the public RLHF-V subset with preference pairs from your own VLM application.
Add a small evaluation harness that compares base and adapter outputs on fixed prompts.
Try the smaller SmolVLM2 variants when iteration speed matters more than peak quality.
Publish the LoRA adapter with a model card that describes the preference data, intended use, and known limitations.