Rules Mapping Issue
The Accessibility Conformance Testing (ACT) Task Force is currently at an impasse. This page summarizes the problem and proposes some potential ways forward. It is intended to support the discussion and induce potential solutions to the issue.
Background
The (Draft) ACT Rules Format specification allows rules for several types of checks, such as the following:
- (R1) Example Rule 1: conformance requirement, partial coverage
- Description: "images must have an accessible name or be marked as decorative"
- This rule is a partial check for WCAG SC 1.1.1 Text Alternatives
- When this rule fails, the associated SC has not been met
- When the rule passes or is inapplicable, further testing is needed to determine if the associated SC has been met or not met
- (R2) Example Rule 2: conformance requirement, full coverage
- Description: "no keyboard trap"
- This rule is a complete check for WCAG SC 2.1.2 No Keyboard Trap
- When this rule fails, the associated SC has not been met
- When the rule passes or is inapplicable, the associated SC has been met
- (R3) Example Rule 3: Sufficient Technique, partial coverage
- Description: "using alt attributes on img elements"
- This rule is a check for Sufficient Technique H37
- It relates to WCAG SC 1.1.1 Text Alternatives
- When the rule passes, the related SC has been met
- When this rule fails or is inapplicable, further testing is needed to determine if the related SC has been met or not met
- (R4) Example Rule 4: Advisory Technique, partial coverage
- Description: "Using a 3:1 contrast ratio or higher as the default presentation"
- This rule is a check for an Advisory Technique (future link)
- It relates to WCAG SC 1.4.3 Contrast (Minimum)
- When the rule passes, the related SC has been met
- When this rule fails or is inapplicable, further testing is needed to determine if the related SC has been met or not met
- (R5) Example Rule 5: Advisory Technique, no coverage
- Description: "positioning labels to maximize predictability of relationships"
- This rule is a check for Advisory Technique G162
- It relates to WCAG SC 1.3.1 Info and Relationships
- When the rule passes, fails, or is inapplicable, further testing is needed to determine if the related SC has been met or not met
Problem Description
The latest ACT Rules Format Editor Draft defines "Accessibility Requirements" with:
- "Accessibility Requirements List": to list the accessibility requirements that the rule checks for
- "Accessibility Requirements Mapping": to express how the result of the rule relates to the listed requirements
Problem 1
According to this latest editor draft of 15 October 2018, the relationships between the rule and the associated requirement can have any combination. For example, if the rule fails then the requirement is not met or further testing is needed. There are no constraints, including for combinations that do not make sense, such as if the rule fails then the requirement is met.
Up until the previous draft, the relationship was constrained to only "fail on fail". That is, if the rule fails then the requirement is not met. What happens on pass or inapplicable can be different depending on the rule, but the fail condition always means the requirement was not met. This constraint was dropped in the latest editor draft, which did not occur to some.
Arguments raised in favor of maintaining the "fail on fail" constraint include ensuring good rule design. This is in the context of conformance requirements, such as example rules R1 and R2 outlined above. It is argued that when a rule checks for a conformance requirement, failing the rule must always mean failing to meet the requirement.
Problem 2
Currently there is no consistent way to refer to checks that are not conformance requirements, such as Techniques (including Sufficient and Advisory, as illustrated by example rules R3, R4, and R5). In some contexts, these checks could be "accessibility requirements", for example when an organizational policy or standard requires such a check. However, in other context, these checks may not be "accessibility requirements" but rather "best practices".
The request is to have a mechanism, similar to "Accessibility Requirements", to refer to the checks that rule is testing for, and to state the relationship (mapping) between the rule and that check. For example, that the check is a Technique H37, if the rule fails then the Technique was not met, and so forth. This could be a replica of "Accessibility Requirements", just with a clearly different name to distinguish rules that check for conformance from others that check best practices.
Problem 3
Following Problem 2, sometimes there is a relationship between the best practice check and a conformance requirement. For example, example rules R3, R4, and R5 outlined above relate to WCAG Success Criteria. In some cases, best practices exceed conformance requirements, such as in example rules R3 and R4. When these rules pass, the conformance requirement is met. However, when the rule fails, the conformance requirement may still be met or not met. In other cases, such as example rule R5, the best practice does not exceed the conformance requirement so that neither passing or failing the rule concludes meeting or not meeting the conformance requirement.
The request is to express a relationship between the outcome of best practice rule and the conformance requirement that they relate to. However, this contradicts the request raised in Problem 1, of requiring only "fail on fail" relationships between rules and conformance requirements. Example rules R3, R4, and R5 do not support such a fail-on-fail relationship.
Potential Options
Some of the options include the following:
- (O1) Option 1: Roll back
- Revert to the Working Draft of 7 July 2018
- Has only "Accessibility Requirements"
- Requires fail-on-fail relationship
- Does not specify other relationships
- No structure for referring to best practices
- (O2) Option 2: Freeze
- Maintain the Editor Draft of 15 October 2018
- Has "Accessibility Requirements List" and "Accessibility Requirements Mapping"
- Allows all types of relationships
- No structure for referring to best practices
- (O3) Option 3: Constrain
- Maintain the Editor Draft of 15 October 2018
- Has "Accessibility Requirements List" and "Accessibility Requirements Mapping"
- Require fail-on-fail relationship
- No structure for referring to best practices
- (O4) Option 4: Duplicate
- Build on the Editor Draft of 15 October 2018
- Has "Accessibility Requirements List" and "Accessibility Requirements Mapping"
- Add "Other References List" and "Other References Mapping"
- Allow all types of relationships
- (O5) Option 5: Duplicate and Constrain
- Build on the Editor Draft of 15 October 2018
- Has "Accessibility Requirements List" and "Accessibility Requirements Mapping"
- Add "Other References List" and "Other References Mapping"
- Require fail-on-fail relationship for accessibility requirements
- (O6) Option 6: ACT Rules maps to testable statements without distinguishing between requirements (normative) and recommendations
- (Added by Anne, based on Option 2)
- Has "Outcome mapping" - and maybe something like "Related recommendation, but with no outcome mapping" (or this could live in a "Background" section or the like)
- Allows all types of relationships
- NOTE: Idea here is that ACT Rules tell which testable statement in an accessibility standard the Rule is testing, but without specifying what the status of this testable statement is. It is the job of the standard to tell what testable statements are normative requirements and which are something else. The job of the ACT Rule is to tell which statement it is testing.
Options Examplified
The following table shows how the options O1-O5 would apply to the example rules R1-R5 from above.
| R1 | R2 | R3 | R4 | R5 | |
|---|---|---|---|---|---|
| O1 |
|
|
|
|
|
| O2 |
|
|
|
|
|
| O3 |
|
|
|
|
|
| O4 |
|
|
|
|
|
| O5 |
|
|
|
|
|
| O6 |
Outcome mapping: Failed:
|
Outcome mapping: Failed:
Passed:
Inapplicable:
|
Outcome mapping: Failed:
Passed:
|
Outcome mapping: Passed:
|
Outcome mapping: Failed:
Passed:
Related, but with no outcome mapping: (NOTE: could be somewhere else)
} |