Apple researchers have published details of SQUIRE, an AI-powered testing tool that helped developers test and refine interface ideas with more control than traditional AI coding tools. Here are the details.
An interesting way of AI-powered interface prototyping
In a new paper titled SQUIRE: Interactive UI Authoring via Slot QUery Intermediate REpresentations, a group of Apple developers proposes a new approach to AI-generated interfaces.
As they explain, natural language brings more flexibility to many aspects of the development process, but it also brings two key challenges:
First, natural language itself can be confusing, making it difficult for developers to clearly communicate their intentions. Second, the model may respond unpredictably, requiring the developer to iterate through trial and error to correct any unwanted changes.
This is where SQUIRE comes in. It is a visualization that allows developers to build and refine UI prototypes step by step, with clear control of results.
From the research:
In SQUIRE, users start a project by providing a command that defines their UI goals, along with a sample dataset containing information for SQUIRE to use as a reference. Users then build the UI as a tree of components in a top-down fashion by telling SQUIRE to fill holes that represent functionality that is not yet expected. In response to this type of request, SQUIRE generates a list of suitable alternatives, each pointed directly to the target port in the unfinished UI. Clicking on one quickly updates the live rendering preview and source code, making it easy to see the difference. The user can also apply target requests to modify the appearance of certain areas of the UI, with the assurance that no code outside the target scope will be changed. In response to this type of request, SQUIRE generates ephemeral controls that allow the user to apply semantically relevant changes quickly and without re-information. In all cases, LLM acts as a friend, presenting relevant decisions for the user to evaluate, but leaving the user and the agency to accept or reject its suggestions.
In other words, natural language prompts were still the way developers interacted with SQUIRE, but instead of touching the entire interface at once, each prompt was tied to a specific part of the UI.
Based on observations with 11 developers using SQUIRE to create interface prototypes, the researchers found that participants were able to test and iterate on different UI designs with a strong sense of control, while rating the system highly on usability and overall satisfaction.
Additionally, they realized that this added sense of control made engineers more comfortable exploring approaches they might not have tried otherwise, since changes were easier to make, predict, and undo.
From research and:
Through data collected from a user survey of 11 leading developers, we find that (1) the SQUIRE interaction encouraged participants to test more frequently, rather than simply using SQUIRE as a code accelerator, (2) participants felt encouraged to take risks when making changes, knowing that the results of making unusual decisions could always be reversed without the associated conflict (3) when making changes, and (4) participants were generally happy with the quality of code and graphics displayed by the program.
SQUIRE under the hood
Rather than generating code to interact directly with user commands, SQUIRE first creates its own intermediate representation, called SquireIR, which models the UI as a tree of objects with named spaces that can be populated over time, as in the example below:
That structure may also include placeholders for undefined components (eg, a button label, image, or content element), as well as many other possible UI methods. For example, it may represent the same content as a list or grid.
From there, SQUIRE translates that presentation into code using HTML, CSS, and JavaScript, with web parts handling the final UI design.
Another important feature of SQUIRE is the way it handles changes.
If a developer requests to modify a button or modify a layout, only that part is updated, while everything else remains untouched.
According to the researchers, this helps avoid the trial-and-error pitfalls seen in many AI coding tools due to the unpredictability of LLMs, where the model can make changes beyond what the developer intended.
This feature also allows SQUIRE to propose multiple options at each step, so developers can quickly compare different versions without losing their previous work.
Unlike many technical papers, this study does not go into details about training models, structures, or data. The researchers note that SQUIRE is powered by OpenAI’s GPT-4o, but the focus of the paper is on the system design and interaction model.
SQUIRE is not widely available, and its use was limited to the 11 engineers who participated in the study. However, it’s not hard to imagine how something like this could be done in future versions of Xcode, or other Apple development tools.
To learn more about SQUIRE, follow this link.
It’s worth checking out on Amazon
FTC: We use auto affiliate links to earn income. More.
