Investigating a failure of axe-core's link-in-text-block rule

The combination of axe-core, the accessibility testing tool and Playwright, the test automation library can be really useful. Both are free, open source tools and while automated accessibility testing has limitations, I often advise teams who take accessibility seriously to consider integrating axe-core into their automated tests, using Playwright, as a guardrail against accidentally creating an easy to identify, easy to fix accessibility bug.

Today, I wanted to share something that confused me. This combination of tools means that when you experience an unexpected behavior, it's sometimes hard to tell; is this a bug with one tool, a bug with the other tool, or is it some quirk about how they work together?

In this case, after the size of an element at the top of the page changed axe started raising the “link-in-text-block” rule.

This rule maps closely to WCAG criterion 1.4.1: Use of color. Typically, this criterion tells us that we shouldn't use color alone to distinguish things. It's not okay, for example, to say required fields are the ones with blue labels because we don't know whether the people using our service can perceive every color, or whether maybe they've overridden the colors on their particular device.

The axe rule only concerns links. If links are always underlined, everyone who is able to see the page can easily tell what's a link and what's not. Often the underline has been removed for esthetic reasons, and this rule checks whether links either have an underline or if the underline has been removed, whether the difference in the luminance of the colors,

So the contrast ratio between the colors of the text and the link is at least 3 to 1. The page in this case did have the links underlined. So I wondered if this issue was actually a false positive. When I tried to figure out what was going on, I created two almost empty pages with links and some other bits of text in, and I tried testing them with axe running in Playwright, testing each page works fine. There are no violations. Where it gets interesting is when you try to test one user journey, starting on one page, clicking the link to go to the other page. Now the first page has no violations and everything's okay, but the second page has the failure to make it even more curious.

This failure doesn't even happen in all browsers. Chrome and chromium-based browsers like Edge result in failures, while Safari - absolutely fine.

So what's going on here? I learned two things while I was trying to debug this. The first, and this shouldn't really have been a surprise, is that when you use Playwright’s click method, there is an actual cursor moving to that actual place. Even if the test is running in headless mode you don't see while you're zooming through the page, but there is a cursor there.

Second, axe is running on exactly the state of the page at exactly that instant, including the hover pseudoclass. When step two loads, the cursor is hovering over that link, and the link in this case was styled that the underline would disappear on hover. axe gets the state of the page in that instant, meaning the underline is now hidden.

This also explains the difference in browsers. If you play around for yourself, you'll notice that Chrome based browsers apply the hover effects from the instant the page loads, even if the mouse hasn't moved yet, while, Safari doesn't apply them until the cursor has moved for the first time. This leaves a few important questions. First, is this axe bug?

I think it's probably not because axe is acting on the exact state of the page, so I don't think it needs to take into account whether some styles are only being applied because something is being hovered on. Second is this a WCAG failure? We think axe was correct in finding the issue.

So, does this fail WCAG? I think it actually doesn't. The normative text of WCAG at least doesn't mention things like styles, that are applied in the hover state and in the default state the underline is there. So this seems to not be a failure.

Even if there isn't a WCAG failure, is there an accessibility problem here? This isn't such an easy question because there are many, many kinds of users with many potential kinds of disabilities. I can't come up with a serious one off the top of my head. If the user navigates using the mouse, then the appearance and disappearance of the underline, as well as the change in the cursor when they hover over the link should make it clear that this is interactive and this is a link. Perhaps there's some risk if a user has zoomed in a very long way and doesn't notice the change in underline style. But I'm not sure. I’d flag this as a theoretical issue, but not one that is a clear showstopper for anyone.

The final question is, is there something we could do better here? If we just want to fix the issue, telling Playwright to just move the mouse somewhere that there isn't something to hover above will fix this issue, but we can actually do something even better here, and we could fix the theoretical issue at the same time. One way to do this would be to swap from hiding the underline on hover to just keeping the underline there, but making it a little thicker. That keeps the visual - that keeps the notice visual change in the link style on hover, but it stops getting into a situation where it might become less clear that this is a link.