Demonstration of AI Powered Accessibility Auditing
This page contains a video recording of the presentation made during the breakout session, along with a transcript. Video captions and transcript were automatically generated and may not properly translate the speaker's speech. Please use GitHub to suggest corrections.
Table of contents
See also:
Video
Transcript
Neha Jadhav: We have tried to use AI
… to reduce this manual intervention.
… This is our POC of a browser extension to get most of the content from the browser. We have implemented it in this format. And this is our Alpha version.
… So let us just deep dive into it.
… Before proceeding, let me just give you a small philosophy behind developing or opting for this model is that we need simplification automation.
… We know how much manual effort, still, we have to put into auditing web sites. And to reduce that we are trying to use AI and try to understand the context, try to understand the content as much as we can with the help of AI, and then try to
… give results based on that.
… We have also tried to implement it in one or 2 languages to start with, and to see how it is affecting our results.
California A: That's something really cool that I didn't mention that we've also done with. So we can localize the language of this tool with the use of AI.
… And one of the one of the big issues with accessibility around the world is that it's very Western based in terms of the characters of the language in terms of the terminology and stuff like this. Translating it into, say, Hindi or Portuguese is a big, big problem.
… How many tools out there are available in Hindi, one of the biggest outsourcing languages we have in tech? India is a huge outsourcer of technology services, yet we have almost no accessibility tools available to them in their native language to help them understand these results. And not that they couldn't understand them in English, but we all know that when you try to translate things over into other languages, you lose contact very easily.
… So one of the biggest things that we try to do is leverage AI to automatically localize the delivery output and input of this tool. And not only can it do it for the user, getting the results and testing or whatever, but we can also use AI to test multiple languages on a page at the exact same time.
… Anyway, please go to your next.
Neha Jadhav: Yes, so one of the rule which we initially, and that was our 1st rule which we implemented with AI, was the abbreviation, and when we saw the result all of us were very excited to see that it was working. Let me just ignore the theoretical thing, and ket me just directly dive into the presentation of AI and show you how it is detecting abbreviation.
… First, as it is an extension, I try to load the extension
… and I click on the tab.
California A: Okay, let's pause very quick. This is our tool. It is named Ottitor. It's a play on words with the word auditor. So it's an audit tool.
… And we've got a logo image of a blue and tan otter using a laptop on its belly. And we figured out having a mascot wouldn't hurt.
… So it's our tool. And what we try to do is we try to to design this so that it's point at things. Only the exact information and features you need at the time that you need it will be available to the user and presented in the user interface to lower problem flow, to make things easier to understand, to lean extremely.
… We try to make it engaging and very concise.
Neha Jadhav: Yes, of course, when we have [inaudible] we can connect to that thing
… immediately, and we feel someone is helping us throughout the process. We have otter with us.
… Let me just sign in into the system.
… I'm selecting advanced AI rules which will help us to analyze all the rules.
… And I have started that audit.
… So, depending on the size of the website, the content present in the website, all the contents will be broken down into chunks, and then they'll send it to the AI model
… to analyze and to get the results back.
… Now, in this, I have specifically added examples for abbreviation in different languages.
… I have received the results. Abbreviations are coming under triple A I have selected.
… And now
… I'll just select the abbreviation
… and I'll see
… the result.
California A: So you run run your testing. What happens is we have an audit results screen that pops up, and on the left hand side we've got the number of issues that are final,
… you can select it by the name of the issue. The top left hand side of the header content block, we've got total number of issues in a sort of a pie chart. We've got issues broken down the number by level by success criteria, by WCAG version. So we get multiple sorting options and when it's clean and concise on the left hand side and along this page
… And on the right hand side, it tells you the success criteria number, which you can also sort by.
… It gives you the the name of the issue that has been found, or issues that have been found, and the exact chunk of code over the issues. You don't do anything on this screen other than see what issues have been found on your site. If you want to go ahead and work on those issues, you click the fix me button
… which takes you to another screen that will allow you to remediate that code, inline, right there, and test it again to see if it passes. So again, kind of use, try to get clean, concise, and easy to manage. So go ahead and please finish describing.
Neha Jadhav: As you can see the sentence present in the website.
… It has WHO and USA, both, as in the abbreviation format.
… WHO is we have highlighted, the AI is also detecting that WHO is not expanded in this sentence, whereas U.S.A., United States of America, is expanded. So both both these things we have highlighted in the results.
… stating the actual reason, present the website. This is one of the example.
… Another example is this, Mine-Resistant Ambus Protected.
… And let me show something in different language now.
California A: Think it looks beautiful. I'm sorry, but I love this design. I'm very happy with what we have.
Neha Jadhav: I'm not sure but I think this is French, so TDM is one of the abbreviation which is present in French, and this is the expanded format offer.
… This is how our abbreviation is working right now. We have tested it on couple of live sites, but, as David has already mentioned, some of the time it stops in between because of the lot of data and number of requests and number of tokens are failing.
… Hopefully, we will have solution. But yes, it is working. If we try to break down the larger content into smaller one, it works fine.
California A: This is an Alpha version. We're working through all of that. We will have a Beta version by...
… What month are we in? By the end of October. So we will have a Beta version release of this by the end of October, and we will have all this stuff sorted out right now. We're playing with things, breaking things, seeing how they work if we like them. And then fixing this.
Neha Jadhav: This is about one of the rule. Now let me try to take you to another rule
… where we have tried to insert some of the idioms we have there, we have tried to insert in different languages.
… Let us see how it works.
… Again, I'm starting Ottitor extension. I open it.
… I sign it into.
… and I run the audit.
… Again, depending on the size of the data that we have, it will take time.
… To save time. We have tried to reduce the size of the data that we are reduced the size of the data in the example HTML files.
… Here it is telling us about 2 rules. One is the link purpose, and another one is the unusual words, idioms that we have used.
… Now, first, let us look at the unusual words that are present in the website.
… Like in this.
… If you really want to see
… what is the result, just select
… it and see the results.
… When I click on the result, it is stating what is the difficult word or a phrase which is present in the website.
… For example, staying ahead of the curve", it says that maintaining a position of the progress in the competitive, so the meaning.
… If it is present in the website or not, it is telling us it is not defined.
… on the ball", it is also not defined, whereas "dog-eat-dog,
… that is defined and it means this.
… This is another example. We have sent the data
… along in the website. What are the difficult worlds? Those are being written from the AI model.
… And I'm very, very happy. This works because I know whenever we are reading any content through the website, sometimes, it happens that we do not understand what is present in it. If it is a secondary language for you
… or if it is a very different context, when you're reading. These kind of things will help auditor it to make it more simplified for the user who are going to use your website. There are some more. This is
… Chinese, or I don't know the language. Sorry about that.
… And here they're trying to analyze everything which is present in different languages.
California A: And one other thing. I wanted to mention this. So right now, we're focused on using AI to find accessibility issues. And we chose to use the word issue instead of violation
… because we want to make this sort of... We don't want to turn users off in terms of thinking Oh, I've got all these problems, I did all this wrong". Mental health kind of things, and appreciate inquiry type of approach, we call these issues.
… But also we want, and that's just a little side step there... But what I really wanted to say is that right now we're using AI for the issues part of it. But we are actually only implementing an AI kind of sort of suggestion button as well. You find these issues, you go to remediate the code, but you're like, Hmm!
… How many ways can I do this? How many ways can I make this better or fix this? I know we can offer some examples. Can AI offer me other examples? Right? So we're going to be implementing an AI to get more examples on how this issue could possibly be fixed in different ways, maybe more innovative or disruptive ways. Who knows? But that's going to be another component we are implementing.
Neha Jadhav: Yes, there can be multiple permutation combination with the results that AI will be providing us after finding issues on the website. Let's see how we can integrate all these things. We have tried with couple of
… things to get the answer from AI. Sometimes it is failing, sometimes it is passing, but detection part is pretty much
… to the point. It is not missing on anything.
… Let us move. We have one more example.
… Whenever we have some links on the website, those links are making sense to the user or not, that is being tested
… in this example. This is
… the 3rd rule which we have implemented here.
California A: And while Ottitor is thinking. Oh, there we go. Okay, never mind.
Neha Jadhav: In the link, we have 2 links in the example, we have 2 links which one link makes sense and another link doesn't make sense.
… So it has detected click here" as the link which is not making sense.
… But here we have to improve more
… where we will try to give more context to the AI model and get the correct answer to the link. That is what we are thinking.
… But it is not yet implemented. But yes, this is how we have detected the links which does not make sense, or the text is not giving any meaningful
… display on the screen.
California A: Can you go back to the other link a little bit.
… It should show us the information.
… Then click your code on the results.
Neha Jadhav: On the results. Okay.
… For the unusual words, or for the link purpose.
California A: Okay. In our tool, this is how this is gonna work. Right here, it says a", "href", a". Click here. So you've got the tip, the line of the code right then and there. This link is ambiguous. You need to fix it, or whatever. All you will need to do when utilizing this tool is change the click here, just backspace, delete delete delete, click, give it a proper name, test it again, it passes,
… log that corrected issue and export it to your github, or whatever issue tracker you use to plug back into your site. That's how simple and easy this is going to be once we launch the Beta version.
… I'm really really proud of how we've thought everything through with those tools, and well, maybe not everything, I'm sure people will come up with some things that we could do better, or that we have forgotten, but I think we've gotten a pretty thorough
… job done with with what we've created.
… Is there anything more that we wanted to talk about this?
Neha Jadhav: Not about the demo part, but yes, definitely, as you mentioned at the starting of the presentation that we are also analyzing the screen recordings.
… What we are trying to do is that we are trying to record
… the session by taking permission from the user and trying to analyze how the website is being displayed to the user, how the audio files are being displayed, how the video files are being displayed. For the video transcript is added or not. Closed captions are working or not.
… And on this, when you record this screen, you also have visual representation of the data so visually, if, everything is making sense, if it is not too much cluttered. So all these rules, which we think that
… they can be implemented or tested by manual intervention only,
… We are trying to analyze that data with the help of AI. And let's see how it goes ahead. Because we have just started analyzing this data, we have not yet mapped this data with any of the rules to understand or to show it to you.
… Once it is done, we will be able to. We have also tried to simulate
… lot of actions from the keyboard and from the mouse, and try to integrate everything
… to create a simulated users action on the screen. So let's see how it goes.
California A: If you think Neha is a little bit too humble, she says try, but I think we've pretty much done that, we just haven't plugged in into our concept. And they work. Is that correct? ...
Neha Jadhav: Sorry. David, your voice is breaking somewhat.
California A: Oh, I'm sorry. Yeah. So I was saying, you say that we were trying to do this, but if I remember correctly, we actually tested the proof of concept for these for these tests and- and they work, correct. We've just implemented it.
Neha Jadhav: They are working. But, as I said, we have not written the rule for a particular rule.
… We have not implemented as per the rule. Analyzing is done. We can understand from the data if the transcript is generated or not, or when you record the screen, is the information present on the screen too much, Is the navigation bar correctly located?
… If we have
… landmarks present on the screen. All this analysis is done, but implementation part as per the rules where we can show you the success criteria listed, as we have, as I showed for other rules that is not being done.
California A: That seems like the easy part. But it's awesome. That's really good.
… It looks like in our proposal we meant for this session, we mentioned we've got current implementations of abbreviations, videos and phrases, text from images, localization which we talked about quite a bit, I think, is really cool,
… analyzing links, and we're talking about future limitations right now, reporting.
… Have we covered all of the future implementations, Neha? Or are there others that we should talk about?
Neha Jadhav: No, all the future implementation is covered.
California A: Okay, so this is what we've done, what we've spent the last 2 years working on with my wonderful team under the tutelage of our chief technology officer, Dr. [missed]. We are incredibly excited and proud of this work. Our senior developer is on the call, he's been doing a lot with our team.
… This wasn't an easy task or easy lift. There are a lot of concerning gaps in enterprise offerings with the AI providers.
… And if anybody is involved in OpenAI or Google Gemini, or any other kind of artificial intelligence, you know company, please do think about these things. Think about how to make your APIs scalable for organizations that want to make SaaS products and change the world, help the world.
… That kind of sort of thought process hasn't happened, and it handcuffs us a little bit on what we can do and what other people can do as well. What I would really love to do is, we would love to hear from the audience, questions, thoughts, comments, concerns anybody got anything for us?