The pros & cons of automated accessibility testing

Automated accessibility testing helps businesses work more efficiently. Why wouldn’t you simplify and speed up tasks that can take a long time to complete manually? The stats do speak for themselves:

• Automating manual tasks can save employees up to 6 weeks per year (State of Work Stats 2023, Salesforce)
• Organisations adopting intelligent automation solutions benefit from increased productivity, cost reduction, improved accuracy and better customer experience (Deloitte)
• 31% of business leaders acknowledge that automation software brings cost-saving benefits (Matican Group)

Sounds good so far? We think so too. But are there any problems with automated accessibility testing? Yes, especially when accessibility overlays are concerned, but we’ll cover that later.

The internet is full of conflicting information, and it’s often hard to know what is right or wrong. We hope this blog will give a jargon-free, holistic view of automated accessibility and its pros and cons and inform you about what is best for your business.

Hold on to your hats, there are lots of acronyms and abbreviations in this blog. Skip to the glossary for quick reference of what they mean.

What do we mean when we talk about accessibility?
What is automated accessibility testing?
Pros of automated accessibility testing tools
Cons of automated accessibility testing
The best of both worlds: Combining automated and manual accessibility testing
Conclusion: Is automated testing enough?
Glossary

Accessibility is the practice of making products and services inclusive and usable to everyone, regardless of their abilities. As Alistair Duggin, Head of Accessibility at GDS, puts it, the goal of accessibility is to exclude no one.

At least one in five people in the UK has a disability, long-term illness or impairment, with the number growing for temporary disabilities.

Website accessibility is a legal requirement. From regulations such as PSBAR and the EAA, accessibility impacts everyone. In the UK alone, there are multiple regulations which organisations need to comply with, for example:

• Public Sector Bodies Accessibility Regulations (PSBAR) 2018
• The Equality Act 2010
• UK Equality and Human Rights Commission (EHRC)
• The European Accessibility Act (EAA)

In addition to being a legal requirement, making a website accessible is important because it’s the right thing to do.

Digital accessibility is governed by the World Wide Web Consortium (W3C) to help build a web experience that works for everyone. The W3C are responsible for the Web Content Accessibility Guidelines (WCAG), international standards which provide detailed guidance on making websites accessible.

WCAG guidance is underpinned by the POUR framework. POUR stands for:

• Perceivable
• Operable
• Understandable
• Robust

These are the four principles of accessibility. They help ensure that digital content and user interfaces are accessible to as many people as possible.

With so many principles and standards to meet, automated accessibility tools exist to make managing accessibility audits easier. Website owners can use an automated accessibility tool to run regular audits to improve the web experience for disabled users.

Automated accessibility testing involves using tools to scan web content and find accessibility errors against WCAG criteria. The tools quickly and efficiently spot accessibility issues, freeing up time for remediation efforts.

Automated accessibility testing can help web teams work smarter. Tools like Insytful can find issues that can be programmatically tested, such as:

• Correct colour contrast
• Missing alt attributes on images
• Form fields without labels
• Correct language attributes for screenreaders
• Broken links
• Misspellings

People often think of accessibility overlays and tools as the same thing, but they are not. While both claim to improve website accessibility, accessibility overlays are well known to cause more harm than good.

Accessibility overlays sit on top of your website and try to modify its appearance, in an attempt to make it more accessible. Usually, they are plugins or widgets, but they do not change the website's underlying code. Therefore, they do not address issues at their source.

An accessibility overlay will generally use JavaScript to try to increase font size, change colour contrast, and add alt text. Still, it can only scratch the surface of the fixes required to make a website accessible. A website cannot meet WCAG compliance using an overlay, as it cannot detect the following, to name a few:

• Incorrect heading structure
• If links are accessible
• Unlabeled form fields
• If buttons have an accessible name
• Replace vague link text like 'Click here' with clear, descriptive labels that convey purpose or destination.

While accessibility overlays are advertised as a quick fix to make websites more accessible, they often do the opposite. In recent years, there have been many legal battles with accessibility overlays, such as AccessiBe, AudioEye, and Userway, to name a few.

As the New York Times mentioned, for blind internet users, the fix can often be worse than the flaws.

In contrast, accessibility tools run automated scans of websites, searching for non-compliance with WCAG standards. The key difference is that accessibility tools do not fix the issue; they simply identify it for website owners to make the fix.

The automated checks make it easy for digital teams with large web estates to keep on top of their website accessibility. So they can spend their time on remediation, rather than tracking faults or waiting for them to be reported.

Insytful is an example of an automated accessibility tool. It audits web content, identifying accessibility issues that website owners can fix. Using an accessibility tool is a long-term solution that helps maintain web content accessibility, in addition to manual user testing.

Overlays are:

• A short-term, quick fix.
• Not able to make changes to the source code.
• A layer on top of your existing website that modifies how the website looks.

Accessibility tools are:

• A long-term solution.
• Are able to identify WCAG non-compliance, but do not make changes for you.
• A complementary tool to use alongside manual accessibility auditing.

Now that you know the difference between accessibility overlays and tools, let's understand the pros and cons of automated accessibility testing tools. There are many benefits, including efficiency and speed, cost-effectiveness, consistency, coverage and peace of mind.

• Accessibility tools can scan and identify website accessibility issues much quicker than humans. There is also less chance of human error using automated testing.
• You don’t need to know WCAG inside out, because accessibility tools like Insytful, programmatically check against WCAG 2.2 standards.

• With an automated tool, you have constant eyes on your content. Saving your business from potential fines and legal battles.
• An automated tool saves your team resources; instead of looking for issues, they can resolve them.

• Automated accessibility tools provide a repeatable testing process for digital teams. Users can work their way through sitewide error lists.
• Monthly progress reports will always take place, even when your team are out of the office. Make use of accurate and meaningful updates with monthly or weekly checks.

• Keep up with your growing and evolving web content, with constant site audits. Feel confident that your content is always being checked for accessibility issues.
• Never miss a beat. In addition to manual audits, automated audits provide a joined-up testing process of your web content.

To keep things balanced, we can’t discuss the pros without discussing the cons, too. Like with most things, automated accessibility has drawbacks. For optimal accessibility results, it should be part of a broader process that involves manual human testing.

• Ultimately, automated tools cannot replicate a human interacting with a website. Therefore, they can’t fully interpret context like a disabled user would.

• Technology gets it wrong sometimes, too. Automated tools can miss accessibility problems or report false positives.

• Automated tools work well when looking for simple accessibility issues, such as whether buttons have accessible names. However, accessibility tools can struggle when issues are more nuanced and complex.
• Tools like Insytful suggest when manual testing is required for a thorough accessibility audit.

To get the best results, website owners should combine multiple testing approaches.

We recommend using automated testing tools like Insytful to find and address the low-hanging fruit, such as finding insufficient colour contrast and links that aren’t accessible. Using accessibility tools can help educate teams about common accessibility errors and learn how to avoid the same issues repeatedly.

In addition to automated tests, website owners should conduct frequent in-depth manual tests with disabled users and accessibility experts to understand issues that testing tools cannot identify.

Automated accessibility testing tools are a good foundation for improving website usability. They provide value for money by checking all web pages and documents, and reduce the staff resources required to audit web content frequently and accurately.

For the most thorough accessibility testing, we recommend regular manual audits to cover the gaps that automated accessibility tools cannot. These audits should be conducted in collaboration with disabled users. Although this approach can be more time-consuming and costly, it gives website owners a true understanding of how usable their site is.

To be clear, automated accessibility tools, like Insytful, can help find the majority of accessibility issues that need to be fixed and provide guidance to reduce recurring problems. On the other hand, accessibility overlays hide accessibility issues from web users and are more of a hindrance than a help.

So when it comes to making your website accessible, here are the key points for you to remember:

• Should you use accessibility overlays? No. They are not a long-term solution, and can get your business into more trouble than having no accessibility testing in place at all.
• Should you use automated accessibility tools? Yes, tools like Insytful and manual testing are good approaches to creating an accessible web experience.
• Can you rely on automated accessibility tools alone? Ideally, not. They should be used in addition to manual testing carried out with disabled users who can identify issues that tools cannot.
• Should you only use manual testing? Manual testing alone can be the most comprehensive approach if you have the staff resources and time available.
• We recommend an integrated approach using automated tools like Insytful with manual practices for optimal accessibility.

• GDS - Government Digital Service (GDS) is responsible for setting, leading and delivering the vision for a modern digital government.
• PSBAR - Public Sector Bodies (website and mobile applications) Accessibility Regulations are UK laws that require public sector organisations to make their websites and mobile apps accessible.
• EAA - European Accessibility Act is a directive with the aim of making products and services accessible in the EU. It applies to public and private sector organisations.
• W3C - World Wide Web Consortium is a non profit organisation that develops web standards. The W3C web standards are the building blocks of the web.
• WCAG - Website Content Accessibility Guidelines is a set of international accessibility guidelines that is developed and maintained by the W3C.
• POUR - Perceivable, operable, understandable and robust are the four principles of accessible design outlined in WCAG.