ARIA 1.1 Post-Mortem

From Accessible Rich Internet Applications Working Group

Rec-track status

  • ARIA 1.1 is a Proposed Recommendation.
  • Core AAM 1.1 is a Proposed Recommendation.
  • DPub ARIA 1.0 is a Proposed Recommendation.
  • DPub AAM 1.0 is a Proposed Recommendation.
  • AccName AAM 1.1 is not yet a Candidate Recommendation.

Workflow

  1. Features added to ARIA spec. ARIA spec entered CR late October 2016.
  2. Testable statements for ARIA spec started, but couldn't be fully completed due to many missing mappings.
  3. Missing mappings sought from platform accessibility API owners for Core AAM.
  4. Testable statements for ARIA spec resumed, written by hand, in a wiki.
  5. Testable statements for ARIA spec corrected because many were written incorrectly.
  6. Testable statements for Core AAM generated via quick-and-dirty JavaScript.
  7. Core AAM spec entered CR late September 2017, work stopping primarily to meet publishing deadlines.
  8. Initial results from automated testing obtained for some platforms, revealing quite a few missing implementations.
  9. Bugs filed, but did not result in implementors doing the implementations. And in some cases we received pushback from them.
  10. All test results for ARIA and Core AAM obtained mid-October 2017. (But not all results were immediately usable.)
  11. Both specs entered PR early November 2017.

Consequences of the 1.1 workflow

  • Disconnect between what we want and what implementors want discovered after entering CR.
  • Critical bugs blocking implementation discovered after entering CR.
  • Errors in AAM discovered after entering CR.
  • Platform owners cannot make changes in mappings without risking exiting CR.
  • Milestones slipped multiple times.
  • Many Working Group members unable to contribute after Workflow step 1.
  • Others?

Lessons learned

  • ARIA features, mappings, implementation, and testing (at least) need to happen together.
  • Implementors (user agent and assistive technology) need to become more involved.
  • Using Core AAM as THE means to show implementability of ARIA is problematic.
  • Treating Core AAM just like "any other (static) spec" is problematic.
  • Using the wiki (and especially wiki table markup) for creating testable statements introduced more problems than it solved.
  • Programmatic generation of testable statements from specs saves a lot of time and human resources!
  • Automated testing tools save a lot of time and human resources! -- when they exist and work as expected.
  • Others?

Conclusions (or at least aspirations)

  • IF
    • we adjust our workflow, and
    • we finish and refine our tools for automating test writing and running, and
    • we document and provide training and support to Working Group members, and
    • we find a way to make the Core AAM a non-static, but still normative, spec
  • THEN
    • we'll catch where we lack consensus with implementors before it's "too late", and
    • we'll all be able to contribute towards advancement along the Rec track, and
    • we'll meet our milestones because we'll always be ready to ship, and
    • we'll have specs which reliably reflect what implementors are expected to do
Retrieved from "https://www.w3.org/WAI/ARIA/wiki/index.php?title=ARIA_1.1_Post-Mortem&oldid=219"