Visual Grounding First,
Multimodal In-context Learning Follows

NeurIPS 2026
Minhyuk Seo1*, Minjae Lee2*, Chaeeun Lee2, Wei Lin4, Muhammad Jehanzeb Mirza5,
Tinne Tuytelaars3†, Jonghyun Choi2†
1UC Berkeley, 2Seoul National University, 3KU Leuven, 4Xero, 5MIT CSAIL
* equal contribution  ·  † corresponding authors
Attention maps for multimodal ICL with a generic identifier (monkey) and a personalized identifier (Jaden)
Visual-token attention when predicting the highlighted word (Qwen-VL-3-8B). (a) With a generic identifier such as “monkey”, every model looks at the right region. (b) Once the demonstrations call the monkey “Jaden”, baselines lose the region and fall back to generic words, while ProCoRe stays grounded and answers “Jaden”.

TL;DR

ProCoRe improves multimodal in-context learning without a single ICL-formatted training example. It fixes the underlying bottleneck, weak visual grounding, with a proposition-level contrastive reward on visually contrastive image pairs.

Abstract

LLMs have shown strong in-context learning (ICL) capability, but extending it to Multimodal LLMs (MLLMs) remains challenging. Prior multimodal ICL methods often rely on ICL-specific datasets for additional training or task vector extraction, which can improve performance on the same ICL benchmark used for adaptation. However, they do not generalize well to other benchmarks and often induce forgetting in previously well-solved tasks.

For this reason, we identify visual grounding as the key bottleneck in multimodal ICL: MLLMs often fail to attend to task-relevant visual evidence in demonstrations, instead over-relying on textual cues and producing hallucinated outputs. To address this, we propose ProCoRe to enhance visual grounding through contrastive reinforcement learning on multimodal contrastive data. ProCoRe generates captions for contrastive image pairs, decomposes them into propositions, and optimizes contrastive rewards that encourage alignment with corresponding images while discouraging mismatched ones.

Despite never seeing ICL-formatted training examples, ProCoRe improves average multimodal ICL classification accuracy by 5.6% and captioning ROUGE-L by 13.8% over the off-the-shelf Qwen-VL-3-8B model, outperforming ICL-data-dependent state of the arts across benchmarks. Finally, we introduce PICL, a personalized multimodal ICL captioning benchmark that tests whether MLLMs genuinely understand visual demonstrations rather than relying on pre-trained semantic priors.

Motivation

Training on ICL Data Does Not Fix Multimodal ICL

Methods that adapt to an ICL benchmark do well on that benchmark, and poorly on the next one.

Multimodal ICL needs the model to bind what it sees in each demonstration image to the text beside it, then carry that link over to the query. Recent MLLMs often skip the first half: they lean on the textual cues and imitate the demonstration text. We ask why this happens instead of training on more ICL data.
TrueMICL accuracy of ICL data-dependent methods trained on TrueMICL versus ISEKAI, compared with ProCoRe
TrueMICL accuracy. Blue-shaded bars are baselines trained on TrueMICL itself; red-shaded bars are the same baselines trained on ISEKAI. They beat the off-the-shelf model mostly when the training and test benchmarks match, and drop below it when they differ. ProCoRe uses no ICL data and outperforms both groups.

Three Routes to Better Multimodal ICL

Retrieve and Order Demonstrations

e.g., RICES, VICL, MMICES

Pick demonstrations similar to the query. Sensitive to prompt design and sample order, and often no better than random selection.

Adapt on ICL Datasets

ICL training and task vectors, e.g., STV, MimIC, MTV

Strong on the benchmark used for adaptation, weak on others, and prone to forgetting previously solved tasks such as simple VQA.

ProCoRe: Ground First

Ours

Train visual grounding on contrastive image pairs. ICL improves as a consequence, across benchmarks.

Finding #1: Limited Generalization and Forgetting

ICL data-dependent methods fit their training benchmark. On a different ICL benchmark they often fall below the off-the-shelf model, and ICL training degrades general VQA.

Finding #2: Weak Visual Grounding Is the Bottleneck

MLLMs attend to irrelevant regions instead of the task-relevant evidence, which leads to hallucination. The failure is sharpest for personalized identifiers that pre-training never covered.

Method

ProCoRe: Proposition-level Contrastive Reward

A caption that misses fine detail fits both images of a contrastive pair, so it earns a low reward. Optimizing the reward pushes the model to describe what is actually in each image.

Overview of ProCoRe: image captioning, proposition decomposition, and reward scoring
Overview of ProCoRe on a contrastive pair that differs in one object (fire hydrant vs. parking meter) and shares the rest of the scene.
1 Caption a Contrastive Pair

The MLLM captions two images that differ only in fine-grained content such as object identity or attributes (InpaintCOCO, EQBEN).

2 Decompose into Propositions

GPT-4o-mini splits the two captions into propositions unique to image 1, unique to image 2, and shared by both.

3 Score and Optimize

Each proposition group gets its own reward, and the MLLM is optimized with Group Relative Policy Optimization (GRPO).

Why Propositions Instead of Whole Sentences?

A sentence-level contrastive reward punishes any content the two images share, because describing it raises similarity to the wrong image. The model then learns shortcut captions like “The object is a hydrant” and drops the sidewalk and the sign. ProCoRe applies the contrast only to image-specific propositions and rewards shared propositions separately, so captions stay complete.
R = λI RI + λT RT + λs Rs

Image–Text Reward RI

Each image-specific proposition should match its own image more than the other one, measured by CLIP similarity.

Text–Text Reward RT

Each image-specific proposition should be entailed by its own ground-truth caption and not by the other, measured by an NLI entailment score.

Shared-Content Reward Rs

Shared propositions should be supported by both images and both ground-truth captions, which keeps common scene details in the caption.

Experiments

Quantitative Results

All main comparisons use Qwen-VL-3-8B. For more models, model sizes, and benchmarks, see the paper.

Method ICL train data TrueMICL (shots) ISEKAI (shots)
01248 024812
Off-the-shelf– 27.1144.7948.8355.0362.70 0.1637.9883.6893.7996.42
RICES retrieval– 27.1144.7947.1651.8361.86 0.1636.3181.2190.5392.36
FV task vectorTrueMICL 28.8243.8848.6155.8760.65 0.9631.5878.7893.1997.92
FV task vectorISEKAI 26.4438.7641.9852.0255.18 1.2135.4484.7292.1897.03
MimIC ICL trainingTrueMICL 27.0848.6254.6555.0061.21 3.0234.8681.0385.5192.19
MimIC ICL trainingISEKAI 27.0838.7443.5551.1256.76 0.1644.8783.0294.7494.11
VOLCANO hallucination– 31.4147.7650.6857.4364.12 0.2142.0874.9694.7392.11
ProCoRe (Ours)none 33.0251.4756.1162.2668.08 3.9450.7286.2396.1898.41

Multimodal ICL classification (accuracy), selected rows from Table 2 of the paper: one representative method per baseline category. For methods trained on ICL data, both training datasets are shown. Bold marks the best score in each column.

Where the Gains Come From

Better ICL comes with stronger grounding and no forgetting.

ROUGE-L on IconQA and TextVQA for ICL training-based methods and ProCoRe
General VQA. ICL training-based methods lose performance on simple VQA tasks (IconQA, TextVQA) that the off-the-shelf model already solved. ProCoRe improves them.
Target attention score on PororoSV and StoryStream
Visual grounding. Target attention score (TAS, higher is better) measures how much each generated token attends to its target image region. ProCoRe scores highest.

Ablation

The gain comes from the reward, and the proposition-level reward beats the sentence-level one.

Method MS COCO (shots) PICL (shots)
01248 01234
Off-the-shelf 15.5319.0221.2123.5225.65 24.5025.4024.7828.3029.29
SFT on contrastive pairs 19.4827.9230.1126.8437.62 22.1327.8823.4130.7426.81
Sentence-level contrastive reward 36.0245.7343.2144.1142.58 32.8834.7431.9234.0233.41
ProCoRe (Ours) 39.1246.8846.4146.2646.03 33.5436.1135.8936.7136.76

Captioning ROUGE-L (Table 3 of the paper). Supervised fine-tuning on the same contrastive pairs gives only marginal gains.

BibTeX

@inproceedings{seo2026visual,
  title     = {Visual Grounding First, Multimodal In-context Learning Follows},
  author    = {Seo, Minhyuk and Lee, Minjae and Lee, Chaeeun and Lin, Wei and Mirza, Muhammad Jehanzeb and Tuytelaars, Tinne and Choi, Jonghyun},
  booktitle = {NeurIPS},
  year      = {2026},
}