Prompt and Circumstance

Back at Ability Summit, I moderated a conversation about shifting accessibility left in the age of AI. During that discussion, Yumeng Ma described a problem she’d encountered in her research: there wasn’t a magical prompt that made generated code accessible. The component mattered. The model mattered. And what we chose to measure mattered too.

That research is the subject of a paper I co-authored with Yumeng Ma, Jacob O. Wobbrock, Joe Devon, Michael Fairchild, Jennifer Mankoff, and Venkatesh Potluri: “Prompt and Circumstance: Evaluating Accessibility in LLM-Generated HTML Form Components”, for ASSETS ’26, the ACM SIGACCESS Conference on Computers and Accessibility.

# Looking at the components

If you ask an AI tool to generate an entire page and then evaluate its accessibility, you can learn something about that page. Working out why it turned out that way is harder. Was it the model? The instructions? A particular component it included?

To help separate those factors, we introduced PACE—Prompt Accessibility Controlled Evaluation—a testbed focused on HTML form components. The study covers:

• 58 form components;
• five large language models;
• three system prompts with increasing accessibility guidance;
• three request variants with increasing detail about each component; and
• 10 trials per configuration.

That adds up to 26,100 outputs. The repeated trials matter: one promising response doesn’t tell us how consistently a model will produce something useful.

As the research summary explains, accessibility varied with prompt construction and the surrounding circumstances, including the model, form component, and measure used to assess the output. That makes a broad claim like “this model generates accessible code” difficult to interpret without knowing how someone arrived at it.

# What are we measuring?

One of the points Yumeng raised during our Ability Summit conversation has stuck with me: an automated check can establish that an image has alternative text without establishing that the text is meaningful. Both questions matter, but they aren’t the same question.

Forms give us plenty of reasons to make that distinction too. Can someone identify what a field is asking for? Can they reach and operate a control with their keyboard? If they make a mistake, can they find the error and work out how to fix it? Those are practical questions I want us to keep asking when we evaluate generated interfaces, alongside whatever automated checks we run.

For me, the useful direction here is more specific evaluation. If we want to improve the code these tools generate, we need to understand where it falls short and which changes actually help. Otherwise, we risk treating a better score as an answer to a question the test didn’t ask.

You can read the paper on ACM, find an author-hosted PDF, or explore PACE. My thanks to my co-authors for the work that went into this. I’m looking forward to seeing how others use and build on it.