CoDeC Contamination Detection in Transformers 5.12.1 with Qwen3, Qwen2.5, and Gemma 3

TL;DR

CoDeC detects likely benchmark contamination by comparing a model’s next-token confidence with and without in-distribution context.
In transformers 5.12.1, Qwen3 and Qwen2.5 work cleanly with AutoTokenizer plus AutoModelForCausalLM, while the Gemma 3 path used in this post relies on AutoProcessor plus Gemma3ForConditionalGeneration.
A thin adapter layer is enough to keep one CoDeC implementation working across all three model families.

Abstract

CoDeC, short for Contamination Detection via Context, is a simple idea with a useful payoff: if a model already knows a dataset suspiciously well, adding nearby examples from that same dataset should help less than you would expect. The original notebook version of this workflow mixed the core method with extra benchmark helpers, training imports, and some stale model choices, which made it harder to reuse. This post trims the method back to the essential scoring logic, updates the code for transformers==5.12.1, and shows how to run the same implementation across Qwen3, Qwen2.5, and Gemma 3. By the end, you will have a practical script you can point at your own text dataset or a benchmark question set.

Requirements

Use this lean setup if you want to reproduce the code as written:

python>=3.10
transformers==5.12.1
torch==2.12.1+cu130
datasets
numpy
accelerate

Prerequisites

Basic familiarity with Hugging Face model loading
Comfort with PyTorch tensor shapes and next-token scoring
A benchmark or text dataset whose raw text you want to inspect for contamination

Problem

Benchmark contamination is awkward because it distorts the signal you actually care about. A model can look strong on an evaluation set not because it generalizes well, but because parts of that set or a very similar distribution appeared during training.

That is the right setting for CoDeC. Instead of trying to prove memorization directly, CoDeC asks a narrower question: does the model behave like the target text is already familiar?

Background

CoDeC compares the same target sample in two conditions:

The model scores the target text by itself.
The model scores the same target text after seeing one or more examples from the same dataset.

The intuition is straightforward:

If the dataset is truly novel, nearby context should help the model.
If the dataset was likely seen during training, the model may already assign high confidence without that extra context.

A convenient score is the difference between the mean token log probability of the standalone target and the mean token log probability of the target when it appears after context:

CoDeC delta = mean log p(target | target prefix)
            - mean log p(target | context + target prefix)

If the delta is positive, the model was more confident without context, which is a contamination warning sign. If the delta is negative, the context helped, which is what you would usually expect on unseen data.

Approach

For a modern transformers implementation, three decisions matter more than anything else:

Use raw dataset text for scoring, not chat-wrapped prompts, unless chat formatting is explicitly part of the thing you want to test.
Skip the earliest few tokens when averaging log probabilities, because beginning-of-sequence behavior is often noisier than the middle of the sample.
Keep the scoring code model-agnostic and isolate only the loading and tokenization differences.

That last point is the reason this update is cleaner than the original notebook. The contamination logic itself is small. Most of the complexity comes from getting a consistent input-to-logits path across different model families.

Key Details

Qwen3 and Qwen2.5

For current Qwen checkpoints, the boring path is the correct path:

AutoTokenizer.from_pretrained(...)
AutoModelForCausalLM.from_pretrained(...)

No training helpers are needed, and no notebook scaffolding is needed. For CoDeC, you only need tokenization, a forward pass, and log-prob extraction.

Gemma 3

For instruction-tuned Gemma 3 checkpoints such as google/gemma-3-4b-it, the safest modern path in transformers 5.12.1 is:

AutoProcessor.from_pretrained(...)
Gemma3ForConditionalGeneration.from_pretrained(...)

That gives you a text path today and leaves the door open for multimodal experiments later without rewriting the loader.

Why raw text matters here

Many current examples for instruction-tuned models center around apply_chat_template(). That is correct for chat generation, but CoDeC is not fundamentally a chat task. If your benchmark row is just a question string, score the question string directly. Wrapping it in assistant-role boilerplate changes the token distribution you are trying to measure.

Keep comparisons fair

CoDeC scores are easier to interpret when you compare:

similarly sized models,
similarly tuned models,
and the same text field across runs.

If one run uses raw benchmark questions and another uses formatted QA records with labels, you are no longer measuring the same thing.

Example

The script below keeps the method small and updates the original notebook idea to a current transformers API.

Two notes before you run it:

The Qwen checkpoints below are small public models that work well for smoke testing.
The Gemma entry uses a public tiny-random checkpoint so the loader path is reproducible without gated-model access. Once you have access to a real Gemma 3 checkpoint, swap in google/gemma-3-4b-it. The code path stays the same.

from __future__ import annotations

import random
from dataclasses import dataclass

import numpy as np
import torch
from datasets import load_dataset
from transformers import (
    AutoModelForCausalLM,
    AutoProcessor,
    AutoTokenizer,
    Gemma3ForConditionalGeneration,
)


@dataclass(frozen=True)
class ModelSpec:
    model_name: str
    loader: str  # "causal" or "gemma3"


class TextScorer:
    """Score plain text under a decoder model."""

    def __init__(self, spec: ModelSpec) -> None:
        self.spec = spec

        if spec.loader == "gemma3":
            self.processor = AutoProcessor.from_pretrained(spec.model_name)
            self.tokenizer = self.processor.tokenizer
            self.model = Gemma3ForConditionalGeneration.from_pretrained(
                spec.model_name,
                torch_dtype="auto",
                device_map="auto",
            )
        else:
            self.processor = None
            self.tokenizer = AutoTokenizer.from_pretrained(spec.model_name)
            self.model = AutoModelForCausalLM.from_pretrained(
                spec.model_name,
                torch_dtype="auto",
                device_map="auto",
            )

        if self.tokenizer.pad_token is None and self.tokenizer.eos_token is not None:
            self.tokenizer.pad_token = self.tokenizer.eos_token

        self.model.eval()

    def _encode(self, text: str) -> dict[str, torch.Tensor]:
        if self.processor is not None:
            batch = self.processor(text=text, return_tensors="pt")
        else:
            batch = self.tokenizer(text, return_tensors="pt")

        return {key: value.to(self.model.device) for key, value in batch.items()}

    def token_logprobs(self, text: str) -> np.ndarray:
        """Return next-token log probabilities for the realized token sequence."""
        inputs = self._encode(text)

        with torch.no_grad():
            logits = self.model(**inputs).logits

        log_probs = logits[:, :-1].log_softmax(dim=-1)
        next_token_ids = inputs["input_ids"][:, 1:]
        token_log_probs = log_probs.gather(-1, next_token_ids.unsqueeze(-1)).squeeze(-1)
        return token_log_probs[0].float().cpu().numpy()


def codec_delta(
    scorer: TextScorer,
    target_text: str,
    context_examples: list[str],
    token_offset: int = 5,
) -> float:
    """Positive deltas suggest the target looked easier without context."""
    no_context = scorer.token_logprobs(target_text)
    with_context = scorer.token_logprobs("\n\n".join(context_examples + [target_text]))
    target_from_context = with_context[-len(no_context):]

    if len(no_context) <= token_offset:
        raise ValueError("Target text is too short for stable CoDeC scoring.")

    no_context_mean = no_context[token_offset:].mean()
    with_context_mean = target_from_context[token_offset:].mean()
    return float(no_context_mean - with_context_mean)


def codec_dataset_score(
    scorer: TextScorer,
    dataset: list[str],
    num_context_examples: int = 1,
    token_offset: int = 5,
    seed: int = 42,
) -> float:
    """Return the fraction of samples with positive CoDeC deltas."""
    rng = random.Random(seed)
    contaminated = []

    for index, target_text in enumerate(dataset):
        pool = dataset[:index] + dataset[index + 1 :]
        if not pool:
            continue

        context_count = min(num_context_examples, len(pool))
        context_examples = rng.sample(pool, k=context_count)

        try:
            delta = codec_delta(
                scorer,
                target_text=target_text,
                context_examples=context_examples,
                token_offset=token_offset,
            )
        except ValueError:
            continue

        contaminated.append(delta > 0)

    if not contaminated:
        raise ValueError("No samples were long enough to score.")

    return float(np.mean(contaminated))


def load_gsm8k_questions(limit: int = 64) -> list[str]:
    """Use raw questions rather than prompt-wrapped records."""
    split = load_dataset("openai/gsm8k", "main", split=f"test[:{limit}]")
    return [question.strip() for question in split["question"]]


def main() -> None:
    dataset = load_gsm8k_questions(limit=64)

    model_specs = {
        "qwen3": ModelSpec("Qwen/Qwen3-0.6B", "causal"),
        "qwen2_5": ModelSpec("Qwen/Qwen2.5-0.5B-Instruct", "causal"),
        "gemma3": ModelSpec("tiny-random/gemma-3", "gemma3"),
    }

    for label, spec in model_specs.items():
        scorer = TextScorer(spec)
        score = codec_dataset_score(
            scorer,
            dataset=dataset,
            num_context_examples=1,
            token_offset=5,
            seed=42,
        )
        print(f"{label}: {score:.3f}")

        del scorer
        if torch.cuda.is_available():
            torch.cuda.empty_cache()


if __name__ == "__main__":
    main()

The key update relative to many older notebook examples is that the scoring path is now explicit and minimal:

tokenize or process the raw text,
run one forward pass,
convert logits to log probabilities,
gather the realized next-token probabilities,
and compare the same target segment with and without context.

Failure Modes & Caveats

CoDeC is simple, but that does not mean every positive score is a clean contamination verdict.

Watch for these issues:

Very short samples are noisy. Skip them or raise the token offset.
Chat-tuned models can have different calibration behavior than base models. Compare like with like.
If context plus target text exceeds the model context window, your score becomes hard to interpret.
Benchmark formatting matters. Question text, answer text, and full serialized records can produce very different deltas.
The public tiny-random Gemma checkpoint is only for validating the API path. Its contamination scores are not meaningful.

For real Gemma 3 runs, you will also need access to the gated checkpoint you choose.

Practical Guidance

If you want usable CoDeC results instead of just a working script, keep the setup disciplined:

Start with one context example before increasing num_context_examples.
Use a fixed random seed so you can compare runs across models.
Compare models of roughly similar size before making strong claims.
Prefer raw benchmark question fields over heavily wrapped prompt templates.
Keep your dataset slice fixed while you iterate on model choice and token offsets.

Sanity check

Before trusting the numbers, verify that:

the model loads without trust_remote_code=True,
the script prints one score per model,
changing the dataset actually changes the score,
and replacing the Gemma debug checkpoint with google/gemma-3-4b-it does not require code changes outside model_specs.

Summary

CoDeC is appealing because the method is much smaller than the surrounding folklore about contamination detection. You do not need a heavy training loop or elaborate instrumentation. You need a reliable text-to-logits path, a careful comparison between with-context and without-context scoring, and enough discipline to keep the dataset representation consistent.

For current Hugging Face users, that translates into one practical rule: separate the model-family loader details from the scoring logic. Once you do that, the same CoDeC implementation can cover Qwen3, Qwen2.5, and Gemma 3 cleanly in transformers 5.12.1.

Next Steps

Swap GSM8K for GPQA, MMLU-Pro, or your own text corpus and compare how the score distribution shifts.
Replace the public Gemma debug checkpoint with a real Gemma 3 release once you have access.
Log the raw deltas per sample, not just the binary contamination decision, so you can inspect borderline cases.