Copyright © 2003-2012 W3C ® ( MIT , ERCIM , Keio), All Rights Reserved. W3C liability, trademark, document use rules apply.
The Multimodal Architecture and Interfaces Specification entered the Candidate Recommendation period on 12 January 2012.
The planned date for entering Proposed Recommendation is 14 August 2012. This document summarizes the results from the Multimodal Architecture implementation reports received and describes the process that the Multimodal Working Group followed in preparing the report.
During the CR period, the Working Group carried out the following activities:
Implementers were invited to contribute to the assessment of the W3C Multimodal Architecture and Interfaces 1.0 Specification by submitting implementation reports describing the coverage of their implementations with respect to the test assertions outlined in the table below.
Implementation reports, comments on this document, or requests made for further information were posted to the Working Group's public mailing list www-multimodal@w3.org (archive).
The Multimodal Interaction Working Group established the following entrance criteria for the Proposed Recommendation phase in the Request for CR:
All three of these criteria have been met. A total of 13 implementations were received from 5 organizations, one each from France Telecom, Openstream and JVoiceXML as well as 2 from Deutsche Telekom and 8 from Telecom ParisTech. The testimonials below indicate the broad base of support for the specification.
All of the required features of the Multimodal Architecture had at least two implementations. The only features which did not have enough implementations were those explicitly listed in the draft specification as "at risk of removal due to potential lack of implementations". These were:
Consequently, these two features have been removed from the final specification as stated in the draft version.
The maturity of the specification has been verified by an interoperability test done by Deutsche Telekom, France Telecom and Openstream. Results of this test have been documented and published as a W3C Working Group Note on 24 January 2012.
pass", "fail" or
"not-impl". "not-impl" means the
Multimodal Architecture constituent has not implemented the specific feature required
by a test.The Multimodal Architecture Implementation Report does not cover:
This section contains testimonials on the Multimodal Architecture from the 5 organizations that submitted implementation reports.
Telekom Innovation Laboratories (Deutsche Telekom R&D) develops multimodal user interfaces for future telecommunication services and is pleased to see the Multimodal Architecture and Interfaces specification moving forward. We have implemented a distributed multimodal interaction management system completely relying on W3C standards: an Interaction Manager based SCXML (using Apache Commons SCXML interpreter), a Graphical Modality Component based on HTML and a Voice Modality Component based on CCXML and VoiceXML. EMMA is used for representation of user input throughout the system. All components are integrated using HTTP as described by the Multimodal Architecture and Interfaces specification. These components have been successfully used in the interoperability test activity with other MMI working group members, demonstrating a distributed multimodal application using components delivered by three independent parties.
Implementing multimodal applications requires to write a lot of markup code (e.g. SCXML, HTML, grammars), which has to be synchronized. To reduce complexity in application development we developed a Multimodal Application Builder, a graphical tool to model multimodal applications and to generate most parts of the code (stubs).
Orange Labs (France Telecom R&D) researches and prototypes multimodal user interfaces on mobile devices. We use and promote open standards, and as such are particularly interested in the W3C Multimodal Architecture specification, which allows building multimodal applications out of specialized components delivered by multiple vendors. Speech, touch and other modalities can be managed by constituents hosted on local devices or on the web, all orchestrated into a single multimodal interaction through the Multimodal Architecture. Orange Labs has implemented an Interaction Manager based on the Commons SCXML application engine, available to Modality Components as an HTTP server. While this implementation is primarily designated for internal R&D work, it was also successfully used in interoperability tests with W3C partners, demonstrating a distributed multimodal application running on components delivered by three independent parties and integrated through the proposed standard.
JVoiceXML is a free VoiceXML interpreter for JAVA with an open architecture for custom extensions. Demo implementation platforms are supporting JAVA APIs such as JSAPI and JTAPI.
Openstream, in its ongoing commitment to open and interoperable standards as a W3C Member, leads the development of multimodal platform and context-aware enablement of speech and other modalities in mobile applications. We are proud to be one of the early and active developers of the W3C Multimodal Interaction Architecture. We have implemented a reference architecture for MMI comprising of Interaction Manager(IM) and several modality components(MCs) as part of our context-aware multimodal platform Cue-me. Adopting the MMI architecture and markup languages such as SCXML, InkML, EMMA, etc. enabled us to easily incorporate and inter-operate with several different vendors' products, enabling us to offer our platform and solutions on a broad array of devices and systems.
This report describes the implementation included in the SOA2M project of the MM Group- TSI Department (Institut Telecom-Telecom ParisTech). This research covers abstract multimodal user interfaces in Service Oriented Architectures for pervasive environemment, an particularly in Smart Conference Rooms. Our prototype is an implementation of a multimodal ambient system providing personalized assistance services to different profiles of users. With this prototype we are looking to test the intelligent automatic fusion and fission of modalities. We are interested on using semantics with the W3C Multimodal Architecture specification. The group has implemented a Flex/AIR Interaction Manager with an SCXML engine, available to Modality Components as a semantically annotated service (published as web or Bonjour service), and web-based RIA modality components at different levels: the basic ones (Pointer, Selector and Graphics IN/OUT) and the more complex (a voice synthesizer and a Carousel).
The aim of this section is to describe the range of test assertions developed for the Multimodal Architecture 1.0 Specification and summarize the results from the implementation reports received in the CR period. The table lists all the assertions that were derived from the Multimodal Architecture 1.0 Specification.
The Assert ID column uniquely identifies the assertion. The Feature column indicates the specific elements or attributes which the test assertion applies to. The Spec column identifies the section of the Multimodal Architecture Specification from which the assertion was derived. The Semantics column specifies the semantics of the feature or the constraint which must be met. The Result column will be annotated with the number of 'pass', 'fail', and 'not implemented' (P/F/NI) in the set of implementation reports.
A conforming Multimodal Achitecture implementation which uses XML to represent MMI Lifecycle Events must utilize documents that successfully validate with respect to the MMI Lifecycle Event XML Schema.
| Spec | Assert ID | Feature | Semantics | Results | ||
|---|---|---|---|---|---|---|
| P | F | NI | ||||
| 5.2.1 | 144 | The Interaction Manager | All events that the Modality Components generate MUST be delivered to the Interaction Manager. | 12 | 0 | 1 |
| 5.2.1 | 145 | All events that are delivered to Modality Components MUST be sent by the Interaction Manager | 6 | 0 | 7 | |
| 5.2.1 | 87 | A modality component may contain its own interaction manager. | 2 | 0 | 11 | |
| 5.2.4.1 | 118 | The Event Transport Layer | The event delivery mechanism MUST guarantee that events delivered from the same source are delivered in that order. | 4 | 0 | 9 |
| 5.2.4.1 | 211 | The event delivery mechanism MUST report an error if an event can not be delivered. | ||||
| 6 | 89 | Modality Components communicate with the Interaction manager via events. | 12 | 0 | 1 | |
| 6 | 90 | Modality Components communicate with the Interaction manager via asynchronous events. | 12 | 0 | 1 | |
| 6 | 91 | Constituents must be able to send events. | 13 | 0 | 0 | |
| 6 | 92 | Constituents must be able to handle events that are delivered to them asynchronously. | 13 | 0 | 0 | |
| 6.1 | 32 | All lifecycle events must use a common basic concept of 'context'. | 13 | 0 | 0 | |
| 6.1 | 33 | A context must represent a single extended interaction with zero or more users across one or more modality components. | 12 | 0 | 1 | |
| 6.1 | 34 | A context should cover the longest period of interaction over which it would make sense for components to store information. | 13 | 0 | 0 | |
| 6.1.1 | 35 | Context | The Context attribute must be a URI that is unique for the lifetime of the system. | 12 | 1 | 0 |
| 6.1.1 | 36 | All events relating to a given interaction must use the same context URI. | 13 | 0 | 0 | |
| 6.1.1 | 146 | Any two events that use different context URIs must be interpreted as parts of unrelated interactions. | 13 | 0 | 0 | |
| 6.1.2 | 37 | Source | The Source attribute must be a URI representing the address of the sender of the event. | 13 | 0 | 0 |
| 6.1.3 | 38 | Target | The Target attribute must be a URI representing the address of the destination of the event. | 13 | 0 | 0 |
| 6.1.4 | 39 | RequestID | The RequestID attribute must be an identifier that is unique within the given context for each Request/Response pair. | 13 | 0 | 0 |
| 6.1.4 | 40 | For any Request/Response event pair, the RequestID in the Response event must match the RequestID specified in the Request event. | 13 | 0 | 0 | |
| 6.1.5 | 41 | Status | The Status attribute must be either 'success' or 'failure'. | 13 | 0 | 0 |
| 6.1.5 | 42 | The Response event of a Request/Response pair must use the Status field to report whether it succeeded in carrying out the request. | 13 | 0 | 0 | |
| 6.1.6 | 43 | StatusInfo | The Response event of a Request/Response pair MAY use the StatusInfo field to provide additional status information. | 4 | 0 | 9 |
| 6.1.6 | 208 | All constituents MUST be able to process Life-Cycle events that use the StatusInfo field to provide additional status information. | 4 | 0 | 9 | |
| 6.1.7 | 44 | Data | Any event MAY may use the Data field to contain arbitrary data. | 12 | 0 | 1 |
| 6.1.7 | 209 | All constituents MUST be able to process Life-Cycle events that use the Data field to provide arbitrary data. | 12 | 0 | 1 | |
| 6.1.8 | 147 | Confidential | Any event MAY use the Confidential field to indicate whether the contents of this event are confidential. | 0 | 0 | 13 |
| 6.1.8 | 148 | The default value of the Confidential field must be 'false' | 1 | 0 | 12 | |
| 6.1.8 | 149 | If the value of Confidential is 'true', the Interaction Manager and Modality Component implementations MUST not log the information or make it available in any way to third parties unless explicitly instructed to do so by the author of the application. | 0 | 0 | 13 | |
| 6.2.1 | 95 | NewContext | A modality component MAY send a NewContextRequest event to the interaction manager to request that a new context be created. | 11 | 0 | 2 |
| 6.2.1 | 151 | The IM MAY create a new context without a previous NewContextRequest by sending a PrepareRequest containing a new context ID to the Modality Components. | 5 | 0 | 8 | |
| 6.2.1 | 152 | The IM MAY create a new context without a previous NewContextRequest by sending a StartRequest containing a new context ID to the Modality Components. | 4 | 0 | 9 | |
| 6.2.2 | 154 | Prepare | The IM MAY send a PrepareRequest to allow the Modality Components to pre-load markup and prepare to run. | 3 | 1 | 9 |
| 6.2.2 | 157 | The Interaction Manager MAY send multiple PrepareRequest events to a Modality Component for the same Context before sending a StartRequest. | 4 | 0 | 9 | |
| 6.2.2 | 158 | Each PrepareRequest the IM sends to a Modality Component MAY reference a different ContentURL or contain different in-line Content. | 3 | 0 | 10 | |
| 6.2.2.1 | 160 | The IM MAY use the same context value in successive PrepareRequest events when it wishes to execute multiple instances of markup in the same context. | 4 | 0 | 9 | |
| 6.2.2.1 | 161 | The PrepareRequest event MAY contain a ContentURL field with the URL of the content that the Modality Component SHOULD prepare to execute. | 4 | 0 | 9 | |
| 6.2.2.1 | 162 | The PrepareRequest event MAY contain a Content field with inline markup that the Modality Component SHOULD prepare to execute. | 2 | 0 | 11 | |
| 6.2.2.1 | 164 | The IM MAY leave both contentURL and content fields of a PrepareRequest event empty. | 3 | 0 | 10 | |
| 6.2.3 | 168 | The IM MAY include a value in the ContentURL or Content field of the StartRequest event. | 5 | 1 | 7 | |
| 6.2.3.1 | 171 | The IM MAY use the same context value in multiple StartRequest events when it wishes to execute multiple instances of markup in the same context. | 3 | 0 | 10 | |
| 6.2.3.1 | 172 | The StartRequest event MAY contain a ContentURL field with the URL of the content that the Modality Component MUST attempt to execute | 6 | 0 | 7 | |
| 6.2.3.1 | 173 | The StartRequest event MAY contain a Content field with inline markup that the Modality Component MUST attempt to execute. | 3 | 0 | 10 | |
| 6.2.3.1 | 175 | The IM MAY leave both contentURL and content fields of a StartRequest event empty. | 4 | 0 | 9 | |
| 6.2.5.1 | 17 | The IM MAY send a CancelRequest to stop processing in the Modality Component. | 5 | 0 | 8 | |
| 6.2.6.1 | 31 | The IM MAY send a PauseRequest to suspend processing by the Modality Component. | 3 | 0 | 10 | |
| 6.2.7 | 1 | Resume | The IM MAY send the ResumeRequest to resume processing that was paused by a previous PauseRequest. | 3 | 0 | 10 |
| 6.2.8 | 61 | ExtensionNotification | The ExtensionNotification event MAY be generated by the IM to encapsulate application-specific events. | 4 | 0 | 9 |
| 6.2.8 | 188 | The ExtensionNotification event MAY be generated by the Modality Component to encapsulate application-specific events. | 4 | 0 | 9 | |
| 6.2.8 | 210 | All constituents MUST be able to process ExtensionNotification events. | 3 | 0 | 10 | |
| 6.2.10 | 74 | Status | The IM MAY send a StatusRequest event to a MC to inquire about the status of a specific user interaction (context) or the status of the MC itself. | 6 | 0 | 7 |
| 6.2.10 | 79 | The recipient of a StatusRequest event MUST respond with a StatusResponse event. | 13 | 0 | 0 | |
| 6.2.10 | 190 | A MC MAY send a StatusRequest event to the IM to inquire about the status of a specific user interaction (context) or the status of the IM itself. | 5 | 0 | 8 | |
| 6.2.10.1 | 75 | The sender of a StartRequest event MAY use the Context field to specify the context for which the status is requested. | 12 | 0 | 1 | |
| 6.2.10.1 | 78 | The StatusRequest event MUST contain a RequestAutomaticUpdate field which is a boolean value. | 11 | 0 | 2 | |
| 6.2.10.1 | 191 | If the Context field is present in a StatusRequest message, the recipient MUST respond with a StatusResponse message indicating the status of the specified context. | 12 | 0 | 1 | |
| 6.2.10.1 | 192 | If the Context field is not present in a StatusRequest message, the recipient MUST send a StatusResponse message indicating the status of the underlying server. | 9 | 0 | 4 | |
| 6.2.10.1 | 195 | If the RequestAutomaticUpdate field of a StatusRequest message is 'true', the recipient SHOULD send periodic StatusResponse messages without waiting for an additional StatusRequest message. | 9 | 0 | 4 | |
| 6.2.10.1 | 196 | If the RequestAutomaticUpdate field of a StatusRequest message is 'false', the recipient SHOULD send one and only one StatusResponse message in response to this request. | 10 | 0 | 3 | |
| 6.2.10.2 | 81 | The sender of a StartResponse event MAY use the Context field to specify the context for which the status is being returned. | 13 | 0 | 0 | |
| 6.2.10.2 | 194 | The StatusResponse event MUST contain an AutomaticUpdate field which is a boolean value. | 11 | 0 | 2 | |
| 6.2.10.2 | 198 | If the AutomaticUpdate field of a StatusResponse message is 'true', the sender MUST keep sending StatusResponse messages in the future without waiting for another StatusRequest message. | 1 | 0 | 12 | |
| 6.2.10.2 | 199 | If the AutomaticUpdate field of a StatusResponse message is 'false', the sender MUST wait for a subsequent StatusRequest message before sending another StatusResponse message. | 11 | 0 | 2 | |
| 6.2.10.2 | 200 | If the Context field is present in a StatusResponse event, the response MUST represent the status of the specified context. | 12 | 0 | 1 | |
| 6.2.10.2 | 201 | If the Context field is not present in a StatusResponse event, the response MUST represent the status of the underlying server. | 9 | 0 | 4 | |
| 6.2.10.2 | 202 | The Status field of a StatusResponse message is an enumeration of 'Alive' or 'Dead'. | 13 | 0 | 0 | |
| 6.2.10.2 | 203 | If the StatusResponse message specifies a context which is still active and capable of handling new life cycle events, the sender MUST set the Status field to 'Alive' | 12 | 0 | 1 | |
| 6.2.10.2 | 204 | If the StatusResponse message specifies a context which has terminated or is otherwise unable to process new life cycle events, the sender MUST set the Status to 'Dead'. | 12 | 1 | 0 | |
| 6.2.10.2 | 205 | If the StatusResponse message doesn't provide a Context field, and the sender is able to create new contexts, it MUST set the Status to 'Alive'. | 9 | 0 | 4 | |
| 6.2.10.2 | 206 | If the StatusResponse message doesn't provide a Context field, and the sender is unable to create new contexts, it MUST set the Status to 'Dead'. | 10 | 0 | 3 | |
| Spec | Assert ID | Feature | Semantics | Results | ||
|---|---|---|---|---|---|---|
| P | F | NI | ||||
| 6.2 | 150 | The Interaction Manager must support the basic life-cycle events. | 5 | 0 | 8 | |
| 6.2.1 | 96 | When the Interaction Manager receives a NewContextRequest event it MUST respond with a NewContextResponse event. | 5 | 0 | 8 | |
| 6.2.1 | 103 | The NewContextResponse event MUST ONLY be sent in response to the NewContextRequest event. | 12 | 0 | 1 | |
| 6.2.1.2 | 107 | If IM has accepted the NewContextRequest, it MUST set the Status field to Success. | 5 | 0 | 8 | |
| 6.2.1.2 | 108 | If the Status property of the NewContextResponse event is Failure then the context identifier MUST be empty. | 11 | 0 | 2 | |
| 6.2.1.2 | 153 | If the NewContextRequest Status field is set to Success, the IM MUST include a new context identifier. | 5 | 0 | 8 | |
| 6.2.2.1 | 163 | The IM MUST NOT specify both the ContentURL and Content in a single PrepareRequest. | 3 | 0 | 10 | |
| 6.2.3 | 166 | Start | To invoke a modality component, the IM MUST send a StartRequest. | 5 | 0 | 8 |
| 6.2.3 | 174 | The IM MUST NOT specify both the ContentURL and Content in a single StartRequest event. | 5 | 0 | 8 | |
| 6.2.7 | 185 | The IM MUST NOT send the ResumeRequest to a context that is not paused due to a previous PauseRequest. | 2 | 0 | 11 | |
| 6.2.9 | 65 | Clear Context | The IM SHOULD send a ClearContextRequest event to a MC to indicate that the specified context is no longer active and that any resources associated with it may be freed. | 5 | 0 | 8 |
| 6.2.9 | 189 | Once the IM has sent a ClearContextRequest to a Modality Component, it MUST NOT send the Modality Component any more events for that context. | 5 | 0 | 8 | |
| Spec | Assert ID | Feature | Semantics | Results | ||
|---|---|---|---|---|---|---|
| P | F | NI | ||||
| 6.2 | 94 | All Modality Components must support the basic life-cycle events. | 11 | 0 | 2 | |
| 6.2.2 | 155 | Modality Components MUST return a PrepareResponse event in response to a PrepareRequest event. | 9 | 0 | 4 | |
| 6.2.2 | 156 | Modality Components that return a PrepareResponse event with Status of 'Success' SHOULD be ready to run with close to 0 delay upon receipt of the StartRequest. | 7 | 0 | 6 | |
| 6.2.2 | 159 | When it receives multiple PrepareRequests, the Modality Component SHOULD prepare to run any of the specified content. | 7 | 0 | 6 | |
| 6.2.2.1 | 165 | If both contentURL and content of a PrepareRequest are empty, the Modality Component MUST revert to its default behavior. | 7 | 0 | 6 | |
| 6.2.2.1 | 207 | If a MC receives a PrepareRequest containing a new context (without a previous NewContextRequest/Response exchange), it MUST accept the new context and return a PrepareResponse message. | 7 | 0 | 6 | |
| 6.2.3 | 167 | The Modality Component MUST return a StartResponse event in response to a StartRequest event. | 11 | 0 | 2 | |
| 6.2.3 | 169 | If the IM includes a value in the ContentURL or Content field of the StartRequest event, the Modality Component MUST use this value. | 6 | 0 | 7 | |
| 6.2.3 | 170 | If a Modality Component receives a new StartRequest while it is executing a previous one, it MUST either cease execution of the previous StartRequest and begin executing the content specified in the most recent StartRequest, or reject the new StartRequest, returning a StartResponse with status equal to 'failure'. | 7 | 0 | 6 | |
| 6.2.3.1 | 176 | If the IM leaves both contentURL and content of a StartRequest event empty, the Modality Component MUST run the content specified in the most recent PrepareRequest in this context, if there is one. | 6 | 0 | 7 | |
| 6.2.3.1 | 177 | If the Modality Component did not receive a PrepareRequest event prior to receiving a StartRequest event with empty Content and ContentURL fields, it MUST revert to its default behavior. | 7 | 0 | 6 | |
| 6.2.4 | 8 | DoneNotification | If a modality component finishes processing it MUST send a DoneNotification | 8 | 0 | 5 |
| 6.2.5 | 178 | Cancel | A Modality Component MUST return a CancelResponse event in response to a CancelRequest event. | 9 | 0 | 4 |
| 6.2.5 | 184 | A Modality Component that receives a CancelRequest event MUST stop processing and then MUST return a CancelResponse. | 9 | 0 | 4 | |
| 6.2.5.2 | 27 | CancelResponse MUST contain a Status field. | 9 | 0 | 4 | |
| 6.2.6 | 181 | Pause | Modality Components MUST return a PauseResponse once they have paused, or once they determine that they will be unable to pause. | 5 | 0 | 8 |
| 6.2.6.2 | 186 | Implementations that have received a ResumeRequest event even though they haven't paused MUST return a ResumeResponse with a Status of 'success'. | 4 | 0 | 9 | |
| 6.2.7 | 187 | The 'Status' of a ResumeResponse event MUST be 'success' if the implementation has succeeded in resuming processing and MUST be failure otherwise | 4 | 0 | 9 | |
| 6.2.7.1 | 5 | Implementations that have paused MUST attempt to resume processing upon receipt of this event and MUST return a ResumeResponse afterwards. | 4 | 0 | 9 | |
| 6.2.9 | 69 | A MC MUST send a ClearContextResponse event in response to a ClearContextRequest event, even if doesn't take any particular action. | 11 | 0 | 2 | |
The Multimodal Interaction Working Group would like to acknowledge the contributions of all the people who allowed for the realization of this activity.