thotab15/molmo_action_ambiguity

Model Card for Molmo-7B-D Action-Ambiguity Resolver (LoRA)

A LoRA fine-tuned adaptation of Molmo-7B-D for detecting and resolving action ambiguity in natural-language robot instructions. Given a scene image and an instruction, the model decides whether the instruction is ambiguous and, if so, generates a clarifying question; otherwise it outputs "none". It targets three action-ambiguity types: WHICH (selecting among multiple matching objects), WHERE (unspecified placement location), and HOW MUCH (unspecified quantity or degree).

Model Details

Model Description

This model is the ambiguity-detection and clarifying-question generation component of a dialogue system for mobile robots which is used to solve underspecified tasks through dialoge between human and robot. Given a scene image and an instruction, the fine-tuned Molmo-7B-D decides whether the instruction is ambiguous and, if so, generates a clarifying question; otherwise it outputs "none". Unlike prior work that handles only object-reference ambiguity, it resolves action-level ambiguity within a single unified framework.

• Developed by: Thota Bhuvana Chandra
• Model type: Vision-Language Model (LoRA adapter) for action-ambiguity detection and clarifying-question generation
• Language(s) (NLP): English
• Finetuned from model: allenai/Molmo-7B-D-0924

Uses

Direct Use

Takes an image and a natural-language instruction as input, determines whether the instruction is ambiguous, and generates an appropriate clarifying question (or "none" when the instruction is clear). Intended for human-robot dialogue in kitchen, household, and office settings.

Downstream Use

Integrated into a robot pipeline: the robot camera feed and user command enter the system; ambiguous commands trigger a clarifying question that resolves the user's intent before the resolved instruction is passed downstream for object localisation and execution, while clear commands route directly onward.

Out-of-Scope Use

• Has not been validated on a physical robot platform; evaluation was offline on static image-instruction pairs, so noisy perception and real-time latency are untested.
• Supports only a single round of clarification, not multi-turn dialogue.
• Trained and evaluated only on kitchen, household, and office scenes with a limited object vocabulary (fruits, bottles, mugs, cups, glasses, bowls, and similar movable items plus surfaces such as tables and placemats).

Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. Increasing the training-set size could improve recall without affecting precision. Validation on a physical robot platform is recommended before real-world deployment.

How to Get Started with the Model

This repository contains the LoRA adapter (plus the Molmo processor and tokenizer files). At inference you load the base model allenai/Molmo-7B-D-0924, apply this adapter on top, then provide a scene image and a task instruction. The model generates a clarifying question ending in "?" (up to an 80-token limit) when the task is ambiguous, or "none" when it is clear. Temperature-scaled sampling with T = 0.2 is used to balance diversity and determinism.

Requirements

pip install torch transformers peft accelerate einops pillow requests

Molmo uses custom modeling/processing code, so trust_remote_code=True is required. A CUDA GPU with sufficient VRAM for a 7B model is recommended.

Download

You can either let from_pretrained pull the files automatically (shown below), or download the adapter explicitly:

from huggingface_hub import snapshot_download

adapter_path = snapshot_download(repo_id="thotab15/molmo_action_ambiguity")

Inference

import torch
from PIL import Image
from transformers import AutoModelForCausalLM, AutoProcessor, GenerationConfig
from peft import PeftModel

BASE_MODEL = "allenai/Molmo-7B-D-0924"
ADAPTER    = "thotab15/molmo_action_ambiguity"

# Processor (handles both image and text)
processor = AutoProcessor.from_pretrained(
    BASE_MODEL, trust_remote_code=True, torch_dtype="auto", device_map="auto"
)

# Load base model and apply the fine-tuned LoRA adapter
base_model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL, trust_remote_code=True, torch_dtype="auto", device_map="auto"
)
model = PeftModel.from_pretrained(base_model, ADAPTER)
model.eval()

# Load a scene image and the instruction
image = Image.open("scene.jpg").convert("RGB")          # or load from a URL
instruction = "Pick up the orange"

inputs = processor.process(images=[image], text=instruction)
inputs = {k: v.to(model.device).unsqueeze(0) for k, v in inputs.items()}

# Generate the clarifying question (or "none")
with torch.no_grad():
    output = model.generate_from_batch(
        inputs,
        GenerationConfig(max_new_tokens=80, do_sample=True, temperature=0.2),
        tokenizer=processor.tokenizer,
    )

generated_tokens = output[0, inputs["input_ids"].size(1):]
response = processor.tokenizer.decode(generated_tokens, skip_special_tokens=True)
print(response)
# Ambiguous instruction -> e.g. "Which orange should I pick up?"
# Clear instruction      -> "none"

Interpreting the output

• If the response is a question ending in "?", the instruction was ambiguous; pass the question to the user, collect their answer, and route the resolved instruction to your downstream object-localisation and execution stages.
• If the response is "none", the instruction is treated as clear and can be routed directly onward.

Training Details

Training Data

A curated dataset built on the AmbRes framework, extended from object-reference ambiguity to action-level ambiguity. It comprises 40 images and 800 annotations.

Training Procedure

Training Hyperparameters

• Adaptation: LoRA on the query and value matrices of the attention layers
• Rank: r = 128
• Scaling factor: α = 256
• Trainable parameters: ~21 million (0.3% of the total model)
• Optimizer: AdamW
• Learning rate: 1 × 10⁻⁴
• Weight decay: 0.01
• Gradient clipping: 1.0
• Epochs: 3
• Batch size: 4
• Inference sampling temperature: 0.2
• Max generation length: 80 tokens

Evaluation

Testing Data, Factors & Metrics

Testing Data

The test split of the dataset: 400 real-world scenarios (260 ambiguous, 140 unambiguous) from kitchen, office, and household settings, drawn from images disjoint from the training set.

Factors

Results are disaggregated by ambiguity type: WHICH, WHERE, and HOW MUCH.

Metrics

Precision, recall, and F1 for ambiguity classification; task success rate. Task success requires all of: correctly identifying the ambiguity type, generating a clarifying question of the correct type, and returning a valid resolution output after the user's response. For unambiguous instructions, success is determined solely by correct ambiguity-type classification with direct routing to grounding.

Results

Overall performance:

Model	Precision	Recall	F1	Task Success
Finetuned Molmo-7B-D (this model)	0.96	0.65	0.77	0.62

Model Card Author

Thota Bhuvana Chandra

Framework versions

• PEFT 0.13.2