Research projectEmbodied AI · Constrained generation

CLAMP: Constrained Decoding for Vision-Language Embodied Planning

A frozen VLM can see a scene and still plan with objects or actions that the scene does not support. CLAMP turns visual evidence and a supplied symbolic action model into hard constraints that act while the plan is generated.

Tianyi Ma Parisa Kordjamshidi

Michigan State University · Heterogeneous Learning and Reasoning (HLR) Lab

Camera-ready manuscript Repository Dataset BibTeX

Observation-conditioned interface Fig. 01

Camera-ready CLAMP framework: visual grounding and a constraint compiler build token-level and action-level DFAs, while an HMM re-ranks the VLM distribution inside the feasible set.

Observation-conditioned hard masks define what is feasible. HMM lookahead ranks only the choices that remain.

A fluent plan is not necessarily an executable plan.

Constrain the distribution before an invalid action enters the sequence.

01 / Why CLAMP

Seeing is not the same as grounding.

Soft visual conditioning can leave unsupported continuations in the model distribution. Post-hoc checking arrives only after those choices have compounded into a plan.

“red mug”→ pick

Perceptual support

Names an object that is not visible

The generated entity is syntactically plausible but absent from the episode-specific support set.

Unsupported entity

placebefore pick

Action transition

Chooses an action before its preconditions hold

The sequence reads naturally, yet the symbolic world state cannot execute the next step.

Infeasible action

valid JSONunsafe

Safety and goals

Satisfies the format while violating a rule

Grammar alone cannot enforce safety, transition, and goal conditions over the full plan.

Constraint violation

02 / Method

A constraint interface between pixels and plans.

CLAMP keeps the VLM frozen. Visual evidence supplies an episode-specific visibility constraint; syntax and the supplied symbolic action model define the rest of the planning interface.

Observe

Image + instruction

The same multimodal context reaches the planner and the interface builder.

Compile

Episode-specific support

Visible entities, syntax, transitions, and goal conditions are compiled into two runtime recognizers.

Decode

Feasible plan tokens

Hard masks remove invalid continuations; a soft prior ranks the feasible set.

Hard · token level

Syntax ∩ visibility

The token-level DFA intersects the action language with the scene-supported entity vocabulary.

Hard · action level

Reachability within budget

The action-level DFA accepts only grounded actions whose supplied transitions preserve goal reachability.

Soft · feasible set

HMM lookahead

A learned sequence prior re-ranks admissible continuations without overriding any hard constraint.

Guarantee boundary. CLAMP-standard returns FAIL or an accepted plan that satisfies the supplied interface, assuming sound perception, specification, and masks. It does not validate those inputs or downstream physical execution.

03 / Results

Grounding improves where the interface acts.

The evaluations separate visual grounding, rule compliance, and decoder cost instead of folding every outcome into a single success claim.

VLABench · frozen Qwen3-VL-8B · 480 prompts

+5.4

points from hard constraints

Observation-conditioned hard constraints raise weighted-DSL from 28.7 to 34.1; the text HMM reaches 37.1 and the supervised vision HMM reaches 38.7.

Baseline

28.7

+ DFA only

34.1

+ Text HMM

37.1

+ Supervised vision HMM

38.7

The final row uses benchmark skill/image pairs plus per-instance calibration. Because there is no matched no-calibration row for that supervised vision HMM, 37.1→38.7 is a cumulative configuration contrast, not an isolated TTA effect.

InternVL3.5-8B · 474 prompts · matched four-dimension subset

Entity-ID validity

Baseline0.019

CLAMP1.000

Format validity remains 1.000. This matched result supports interface conformance; it is not evidence that CLAMP adds visual knowledge.

SafeAgentBench · legitimate long horizon · CLAMP-policy

Symbolic violations

Baseline0.41

Policy0.05

Safe completion rises from 0.06 to 0.18 with backup-and-retry plus validated forced-action injection. This is not the one-pass CLAMP-standard decoder.

TaPA-60 grounding diagnostic

Grounding improves; plan success does not

Out-of-scene references10.0% → 8.3%

Matched plan success71.7% → 68.3%

This is a grounding diagnostic, not a task-success gain; the benchmark self-judge is also sensitive to harness metadata.

Cross-domain HMM calibration · VH→BEHAVIOR

Conditional log-likelihood

Before7.132

After3.095

A 56.6% reduction after 30K target continuations and three Baum–Welch iterations in 417 seconds (about seven minutes), with the VLM frozen. Planning remains mixed: five dimensions show single-skill collapse.

04 / Qualitative evidence

One scene. Three constraint roles.

This InternVL3.5 measured trace records raw and post-processor probabilities from the same frozen generate call—no illustrative re-scoring.

Four-camera RGB view of a VLABench kitchen scene with a pan and condiment bottles. — RGB observation

The same kitchen scene with entity-ID segmentation labels; barbecue sauce is entity 4 and the pan is entity 2. — Entity-ID mask

Task

“This dish lacks a deep, tangy touch. Maybe adding something could elevate its taste.”

Ground truth: pick(4 = bbq_sauce) → pour(2 = pan_seen)

Unconstrained · failure

Names instead of IDs, then keeps going.

pick("bbq_sauce") → pour("pan_seen") → place("dishes")

Schema 0Exact 0Total 0.222

DFA + HMM · accepted plan

Emits the exact integer-ID plan and stops.

pick(4) → pour(2)

Schema 1Exact 1Total 1.000

t = 0

Open valid JSON

Raw ≈0%CLAMP 78.56%

DFA masks a 92.12% markdown-fence token.

t = 20

Use an integer ID

Raw 3.73%CLAMP 99.997%

DFA removes the quote that would open a string value.

t = 21

Select entity 4

Raw 55.92%CLAMP 55.96%

A uniform HMM bonus preserves the legal digits' ranking.

t = 39

Stop at the goal

Raw 11.91%CLAMP 69.83%

Soft HMM re-ranking shifts probability toward closing.

Evidence boundary: This InternVL3.5 measured trace uses the VLABench non-interactive plan evaluator. The DFA supplies schema compliance, visibility supplies entity support, and the HMM raises STOP from 11.91% to 69.83%. It demonstrates interface compliance and correct termination—not improved semantic recognition, a main Qwen3-VL result, or simulator execution-success.

View the complete measured trace

Plan-match example

Grounded target and skill sequence agree

A representative example from the public 20-episode demonstration release.

Counterexample

The skill is plausible; the target entity is wrong

The failure split preserves cases where planning and downstream execution come apart.

05 / Dataset

Inspect the evidence, not just the score.

The public demonstration release pairs scene inputs, grounded plans, expert references, evaluation records, and multi-camera execution videos.

Open on Hugging Face

20: recorded VLABench episodes
10 + 10: plan-match examples and counterexamples
4: synchronized execution cameras
9: artifact types per episode

RGB inputEntity maskInstructionEntity listCLAMP planGround truthEvaluationVideo

06 / Citation

Use CLAMP in your work.

CLAMP appears in Findings of the Association for Computational Linguistics: EMNLP 2026.

@inproceedings{ma2026clamp,
  title     = {{CLAMP}: Constrained Decoding for Vision-Language Embodied Planning},
  author    = {Ma, Tianyi and Kordjamshidi, Parisa},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2026},
  year      = {2026},
  publisher = {Association for Computational Linguistics}
}

Download .bib