ASIL structured software interaction

Scope and limits

Reproduction level

mechanism demo

Claim under test

a structured observation and typed semantic-action interface can reduce ambiguity and action count relative to screenshot-and-click control.

Excluded claim

this tutorial does not reproduce the paper's 15-application benchmark or its reported success percentages.

Reader, effort, and cost

intermediate Python or TypeScript; 4 to 6 hours; local CPU; no paid model required if a scripted policy is used.

Canonical resources

• Paper: https://arxiv.org/abs/2608.26991 (opens in a new tab)
• Project page: linked from the arXiv record. At review time, the arXiv landing page did not expose a canonical public code repository or dataset, so this tutorial uses a disclosed substitute environment.

Project plan

1. 01Build the substitute task(45 minutes)

   Create a local notes application with ten records and three tasks: create, rename, and move a note. Provide both a rendered HTML interface and a JSON state endpoint. Artifact: fixed task fixtures and expected end states.

2. 02Define interfaces(45 minutes)

   Pixel condition receives screenshots and click/type coordinates. Structured condition receives {objects, properties, availableActions} and may call typed actions such as rename_note(id, title). Keep task information identical and log any extra state exposed by either interface.

3. 03Implement equal policies(60 minutes)

   Use one deterministic planner or the same model and prompt for both conditions. Limit each task to 20 actions and 60 seconds. Reject actions not present in the declared schema.

4. 04Run trials(60 minutes)

   Execute at least 30 randomized task instances per condition across three seeds. Log strict final-state success, action count, latency, invalid actions, and recovery after one deliberately stale observation.

5. 05Analyze(45 minutes)

   Report Wilson intervals for success and bootstrap intervals for action count. Success requires fewer actions without lower strict success under information-equivalent state. Falsification occurs if the advantage disappears after removing privileged fields or if a native API matches the structured layer.

6. 06Cleanup(10 minutes)

   Delete screenshots containing test content and preserve only synthetic fixtures and aggregate metrics.

Controls, failure modes, and safety

Affected topics

Back to the August 28, 2026 briefing