When minor edits
matter

Yasamin Medghalchi1, Milad Yazdani1, Amirhossein Dabiriaghdam1, Moein Heidari1, Mojan Izadkhah1, Zahra Kavian1, Giuseppe Carenini1, Lele Wang1, Dena Shahriari1, Ilker Hacihaliloglu1

1 University of British Columbia, Vancouver, Canada

Clinically plausible prompt rewrites—not typos, not jailbreaks—can flip ultrasound vision–language model predictions. We introduce SonoPromptAttack: LLM edits guided by Monte Carlo Tree Search.

Read the paper Interactive explorer Code Cite
0 % Highest attack success on disease diagnosis (MedGemma-27B)
0 %+ Meaning of the original question kept after edits
0 Medical vision–language models tested (query-only access)
0 % Success with MCTS search vs 39% with beam search
01 The problem

The text interface is the attack surface

Ultrasound Med-VLMs are queried in natural language. In POCUS and decentralized care, prompts are informal, abbreviated, or noisy— shorthand from a busy clinic, not a hand-crafted jailbreak.

Prior safety work often tests obviously unsafe instructions. We study something quieter: meaning-preserving rewrites that a clinician could type without intending harm—and still flip the answer.

//

Not typos only

Character noise is easy to spot. Natural paraphrases are not.

??

Intent preserved

Same diagnostic question. Different wording. Different prediction.

!!

Clinical stakes

Benign ↔ malignant is not a cosmetic error—it changes the care path.

02 Method

Black-box edits that stay clinical

SonoPromptAttack never touches the ultrasound image and never needs gradients. An attacker LLM suggests small, meaning-preserving span replacements—synonyms, light reordering, punctuation—while answer option tokens stay locked.

Invalid proposals are dropped by rule (exact span, English-only, no duplicates). Rejection feedback is fed back so the next round is less wasteful. Surviving edits grow a search tree of candidate questions.

Monte Carlo Tree Search chooses which wording branch to try next, scores each rewrite by how much the target model’s confidence shifts, and continues until the answer flips or the budget runs out (at most 80 model queries and 8 edit steps).

Overview of SonoPromptAttack: a language model proposes prompt edits, filters validate them, tree search selects branches using target-model scores, and a prediction can flip after a short edit sequence.
How SonoPromptAttack works: propose small rewrites, keep only valid ones, and search until the ultrasound model’s answer changes (paper Fig. 1).
03 Results

Attack strength without mangling language

We evaluate on U2-Bench ultrasound questions: disease diagnosis (what is wrong?) and lesion localization (where is it?), across five medical vision–language models. Attack success is counted only when the model was correct before the rewrite.

Interactive explorer Browse 200 recorded successful attacks by proposer, target model, and task.

Disease diagnosis — higher attack success and similarity are better; lower perplexity (more fluent text) is better. Highlighted rows are SonoPromptAttack.

Method MedGemma-4B MedGemma-27B QoQ-Med-7B QoQ-Med-32B LLaVA-Med-7B
SuccessSimilarFluency SuccessSimilarFluency SuccessSimilarFluency SuccessSimilarFluency SuccessSimilarFluency
Random typos 23.596.4161 27.396.5147 28.196.3134 21.995.6270 9.196.4106
Greedy typos 18.698.327.9 26.499.026.6 22.898.656.2 20.397.577.0 29.597.734.4
TextFooler 16.598.830.6 22.999.518.6 17.499.225.8 24.198.638.6 12.299.816.6
DeepWordBug 15.099.038.6 24.599.225.4 17.998.742.8 19.997.950.7 5.999.916.3
Ours · Qwen 7B 27.098.914.5 34.898.914.4 24.498.814.1 27.299.014.8 14.098.911.1
Ours · Qwen 14B 42.998.716.6 52.098.816.8 40.198.616.4 35.298.917.4 15.999.110.9
Ours · Gemma E4B 47.999.118.2 60.999.218.2 45.399.119.9 32.799.118.4 18.099.211.7
Ours · Gemma 12B 38.498.719.4 40.398.917.8 32.498.520.8 26.398.819.5 19.899.311.1

On disease diagnosis, SonoPromptAttack has the highest attack success for four of five models (second on LLaVA-Med-7B) and produces the most fluent rewrites overall.

Vulnerability depends on the body region

Average scores hide big differences by anatomy. Lung questions are often easy to flip (about 47–66% mean success depending on the model). On MedGemma-4B, skin and knee questions are also fragile (~69% and ~71%). PCOS and some thyroid cases hold up much better for certain models. Robustness checks should break results down by anatomy—not only report one overall number.

Bars show mean attack success by anatomy for MedGemma-4B (higher = more vulnerable).

Attack success rate by anatomy and target medical vision-language model. Colored markers show each rewrite model; gray bars show the average across rewrite models.
Disease-diagnosis attack success by anatomy and model: some organs and models are far more sensitive to wording than others (paper figure).
04 Perturbation quality

Stronger attacks without mangled text

Baseline methods often succeed by injecting typos or swapping clinical terms. SonoPromptAttack aims for rewrites a clinician could still read: natural wording, correct grammar, and the same diagnostic intent—checked by an automated language judge.

Method Typical prompt change
Char random radiologist → hre a radiologist · breast → breasd
TextFooler breast ultrasound → torso sound — drifts off the anatomy
Ours synthesize these features → integrate these findings

Across both tasks, SonoPromptAttack ranks best or second-best on quality scores (naturalness, grammar, intent) in most settings—while remaining competitive on attack success and leading on fluency.

Successful prompt-rewriting attacks across six ultrasound anatomies. Each panel shows original and rewritten prompts with prediction flips such as benign to malignant.
Real examples: small wording changes flip the model’s answer across six ultrasound anatomies (target model MedGemma-4B; rewrites proposed by Qwen2.5-14B — paper Fig. 4).
05 Ablation

Why tree search beats greedy climbing

We keep the same rewrite model, filters, and query budget (80 model checks, at most 8 edit steps) and only change how candidates are chosen. The chart below is for lesion localization on LLaVA-Med-7B, with Qwen2.5-7B proposing the rewrites.

Spending the same budget more carefully matters: greedy search gets stuck on weak early edits, while Monte Carlo Tree Search roughly doubles the best baseline.

06 Setup

Models & data

Models we attack

  • MedGemma (4B & 27B)
  • LLaVA-Med (7B)
  • QoQ-Med (7B & 32B)

Models that rewrite prompts

  • Qwen2.5 (7B & 14B)
  • Gemma 4 (E4B & 12B)

Benchmark

  • U2-Bench disease diagnosis (1,911 images)
  • U2-Bench lesion localization (503 images)
  • Breast, thyroid, lung, skin, knee, pancreas, pelvic…

Baselines we compare to

  • Random & greedy character typos
  • DeepWordBug (character noise)
  • TextFooler (word swaps)
07 Paper

arXiv:2603.21047 · cs.CV

When Minor Edits Matter: LLM-Driven Prompt Attack for Medical VLM Robustness in Ultrasound

arXiv PDF arXiv abs Code BibTeX

Takeaways

  • Routine clinical phrasing is enough to stress Med-VLMs—no need for overtly malicious prompts.
  • MCTS-guided LLM edits beat character/word baselines on the effectiveness–quality trade-off.
  • Anatomy and model scale both modulate vulnerability; report both.
  • Future: more US tasks, other modalities, open-ended reports, and robustness-aware defenses.

Cite

@misc{medghalchi2026minoreditsmatterllmdriven,
  title         = {When Minor Edits Matter: LLM-Driven Prompt Attack for Medical VLM Robustness in Ultrasound},
  author        = {Yasamin Medghalchi and Milad Yazdani and Amirhossein Dabiriaghdam and Moein Heidari and Mojan Izadkhah and Zahra Kavian and Giuseppe Carenini and Lele Wang and Dena Shahriari and Ilker Hacihaliloglu},
  year          = {2026},
  eprint        = {2603.21047},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2603.21047}
}