Strictness Is a Feature: Why We Reject What We Don't Understand

Part 5 of "Testing the Way Users Actually Experience the Web" — a tutorial series on @afixt/usecase-runner.

This is a post about a class of bug that test tools are prone to, what it costs, and why usecase-runner is deliberately unforgiving about it.

Consider this step:

- verify: text "Current Page" attribute "aria-current" is "page"

Read it and you know what the author meant: the element showing "Current Page" should carry aria-current="page". Reasonable assertion, clear intent.

Here is what a lenient parser does with it. The text sub-type of verify reads its quoted string, builds a visibility check, and returns. Everything after the closing quote — attribute "aria-current" is "page" — is simply discarded. The step compiles to:

await expect(page.getByText('Current Page')).toBeVisible();

It passes. It passes on pages with aria-current="page". It passes on pages without it. It passes on pages where aria-current is "true", "false", or "banana". And nothing in the output distinguishes that pass from the pass it would have produced had the assertion actually run.

In one real suite, a step like this was the only ARIA assertion in a case titled "ARIA Roles, States, and Properties." It was green. The green was counted as a fix.

There is a second path, too. Unrecognised modifiers on any verb — activate: button "Save" inverts "aria-checked", say — can be pushed onto a warnings array that nothing ever reads. Not the CLI, not the validator, not the runner. A specification that says the processor "MAY report" unknown modifiers as warnings is satisfied to the letter by writing to a list nobody reads.

Here's the argument that decided the fix, recorded in ADR-0007:

A rejected step gets fixed in minutes. A silently ignored one becomes a permanently green assertion that inflates pass counts and is never re-examined, because nothing distinguishes it from the check passing.

A test tool has exactly one job: tell you the truth about whether a condition holds. A test that reads as an assertion and executes as nothing is the most expensive kind of defect a test tool can have — worse than a false failure, because a false failure gets looked at.

So unconsumed input is a parse error. Every verify: sub-type asserts it consumed its whole input. Every action verb rejects a modifier it doesn't know. The error names the offending token, the running version, and the accepted set — so a clause that is merely newer than your installed runner is distinguishable from a typo:

Unexpected trailing input after verify: text — "attribute "aria-current" is "page"".
  in step: verify: text "Current Page" attribute "aria-current" is "page"
  usecase-runner <version> does not accept modifiers on "text"; it would be ignored
  at runtime rather than checked.

The actual fix is to target the element by role so the attribute clause has something to attach to — and then it does what it says:

- verify: link "Current Page" attribute "aria-current" is "page"

The same strictness catches mistakes that would otherwise surface late. A keyboard: step with a literal sentence in it (keyboard: 'Hello world') would compile to a key press that throws at runtime; it fails at validate time instead, with type: named as the verb the author wanted. Green lies are more expensive than a validation error.

Strictness shows up again in attribute assertions, in a place that surprises people.

- verify: button "Show details" attribute "aria-expanded" is "false"

If the button has no aria-expanded attribute at all, this fails. Some users expect it to pass — "the ARIA default for a missing aria-expanded is effectively collapsed, isn't it?"

It isn't. The spec default for aria-expanded is undefined, which means "this element is not expandable." A disclosure button with no aria-expanded tells a screen reader nothing about its state. Absence is a genuine defect, and the assertion is right to fail. The same holds for aria-pressed, aria-checked, aria-selected, and aria-hidden.

But there's a second family of attributes where the opposite is true. aria-invalid, aria-disabled, aria-required, aria-readonly, aria-busy all default to false, and omitting them is the conformant way to say "false." A field that has never been submitted simply has no aria-invalid. So this:

- verify: field "Email" attribute "aria-invalid" is "false"

fails on exactly the markup it was written to bless.

We could have made is resolve ARIA defaults. We deliberately did not, because that would silently weaken every assertion on the first family. Instead, leniency is something you opt into, one assertion at a time, with a predicate that says what it means:

# Equals "false", OR the attribute is omitted — both are conformant here
- verify: field "Email" attribute "aria-invalid" is_or_absent "false"

# The attribute must not be there at all
- verify: button "Save" attribute "aria-hidden" absent

is means is. If you want "is or isn't there," you write that, and a reviewer can see you chose it.

- verify: field "Email" attribute "aria-required" is "true"
- verify: combobox "Country" attribute "aria-activedescendant" present
- verify: button "Save" attribute "title" absent
- verify: field "Email" attribute "aria-invalid" is_or_absent "false"
- verify: button "Open menu" attribute "aria-controls" starts_with "menu-"
- verify:
    listbox "Items" attribute "aria-activedescendant" matches "option-\\d+"
- verify: dialog "Confirm" attribute "aria-describedby" references_existing_id
- verify:
    slider "Volume" attribute "aria-valuenow" within_range_of "aria-valuemin"
    "aria-valuemax"

references_existing_id deserves a mention: for relationship attributes (aria-labelledby, aria-describedby, aria-errormessage, aria-controls) the meaningful question is "does every id in this value resolve to a real element?", not "is it this literal string?" A dangling aria-describedby is a silent failure for the user; this predicate makes it a loud one for you.

Take any existing use case and append nonsense to a step:

- activate: button "Save" with_confidence true

npx usecase-runner validate ./usecases

It will refuse. That refusal is the feature.

Next: select Is Not toggle: Verbs With Contracts