EMNLP 2026

IGG: A Benchmark for Interactive GUI Grounding under Visibility Constraints

Kyeongseon Kim* · Jiyeon Son* · Tae-Hyun Oh
KAIST
*Equal contribution

IGG evaluates whether screenshot-only GUI agents can recover a target's visibility through minimal interaction—and then ground the correct element.

Paper · soon Dataset · soon
601tasks
16mock applications
5atomic actions
7constraint sub-types

Benchmark at a glance

Grounding starts before the click.

The missing middle

Beyond static pixels.
Short of full navigation.

Static GUI grounding assumes the requested element is already visible. Real interfaces violate that assumption: targets may be off-screen, visually ambiguous, hidden by state, occluded, delayed, or moving.

IGG keeps the goal precise—locate and click one target—while adding only the interaction required to make that target observable. Every click is verified online against the target element in the current UI state.

01Interactive grounding, at a glance
Observe · Reveal · Ground
01Observability02Disambiguation 03State control04Temporal control

Screenshot-only target revelation

SCROLLMOVECLICKWAITDRAG
Static GUI Grounding
+reveal interaction
1Viewport Constraint

Reveal by viewport movement

Scroll-based Exploration
2Multi-State Constraint

Reveal by state-changing interaction

Visual DisambiguationHidden State ActivationRemovable Occlusion
3Temporal / Manipulation

Reveal over time or by manipulation

Delayed State ActivationDynamic Target InteractionMovable Occlusion
Human–model gapGrounding Accuracy (%)
HumanGPT-5.5Qwen3-VL
Viewport
90.0
Multi-State
94.4
Temporal &
Manipulation
94.4
Figure 1. Four key capabilities and five atomic actions define three visibility constraint types.

Controlled interaction

Three types.
Seven constraints.

SCROLLMOVEDRAGWAITCLICK
01Viewport Constraint

Reveal by viewport movement

The target is hidden by the current viewport but remains reachable in the same UI state.

  • CT1-1Scroll-based Exploration
02Multi-State Constraint

Reveal by state change

The target or the cue needed to identify it appears only after an interface interaction.

  • CT2-1Visual Disambiguation
  • CT2-2Hidden State Activation
  • CT2-3Removable Occlusion
03Temporal & Manipulation

Reveal over time or by manipulation

The target requires temporal control, tracking a dynamic state, or spatial manipulation.

  • CT3-1Delayed State Activation
  • CT3-2Dynamic Target Interaction
  • CT3-3Movable Occlusion

Dataset

601 tasks across 16 controlled mockups

Seven sub-types span off-screen targets, interface state changes, delayed and dynamic content, and movable occlusions. Every task is replayed to verify that its target is unique and reachable.

27.8%Viewport
40.9%Multi-State
31.3%Temporal & Manipulation
02Task distribution
601 verified tasks
Constraint sub-types 167 Scroll 89 Visual 104 Hidden 53 Removable 70 Delayed 58 Dynamic 60 Movable
Tasks per platform02550
PRODUCTIVITY
42
42
40
35
33
DEV & DESIGN
45
35
32
SOCIAL & MEDIA
49
34
50
39
34
LIFESTYLE
31
30
30
Task distribution by type, subtype, and platform.
ChirpDayplanDevhubDevpadDiscordDocsFigmaGridlyMailoNeverPagelyReelifyShoppingWavrWayfindYouTube

Main results

Visibility recovery is the bottleneck.

Humans solve nearly all tasks, while the strongest open-source baseline reaches only 31.6% GA. Completing the reveal interactions before Qwen3-VL's final click raises its GA from 30.8% to 81.9%.

94.6%Human GA
79.4%GPT-5.5 GA
31.6%Best open-source GA
+51.1Oracle-reveal gain
03The interaction gap
Grounding Accuracy
Human94.6%
GPT-5.579.4%
GUI-Owl 1.5 best open source31.6%
Qwen3-VL30.8%
Oracle reveal30.8% → 81.9%

Completing the required reveal interactions unlocks a +51.1 point gain for Qwen3-VL.

ScrollVisualHiddenRemovableDelayedDynamicMovable
Performance across the seven visibility constraint sub-types.

Citation

Build on IGG-Bench

Please cite our EMNLP 2026 paper. The proceedings identifier will be added after publication.

@inproceedings{iggbench2026,
  author = {Kim, Kyeongseon and Son, Jiyeon and Oh, Tae-Hyun},
  title = {IGG: A Benchmark for Interactive GUI Grounding
           under Visibility Constraints},
  booktitle = {EMNLP},
  year = {2026}
}