Multimodal Interaction Working Group Charter

Scope

Background

The primary goal of this group is to develop W3C Recommendations that enable multimodal interaction with various devices including desktop PCs, mobile phones and less traditional platforms such as cars and intelligent home environments including digital TVs/connected TVs. The standards should be scalable to enable richer capabilities for subsequent generations of multimodal devices.

Users will be able to provide input via speech, handwriting, motion or keystrokes, with output presented via displays, pre-recorded and synthetic speech, audio, and tactile mechanisms such as mobile phone vibrators and Braille strips. Application developers will be able to provide an effective user interface for whichever modes the user selects. To encourage rapid adoption, the same content can be designed for use on both old and new devices. The capability of possible multimodal access depends on the devices used. For example, users of multimodal devices which include not only keypads but also touch panel, microphone and motion sensor can enjoy all the possible modalities, while users of devices with restricted capability prefer simpler and lighter modalities like keypads and voice.Extensibility to new modes of interaction is important, especially considered in the context of the current proliferation of types of sensor input, for example, in biometric identification and medical applications.

Target Audience

The target audience of the Multimodal Interaction Working Group (member only link) are vendors and service providers of multimodal applications, and should include a range of organizations in different industry sectors like:

Mobile and hand-held devices

As a result of increasingly capable networks, devices, and speech recognition technology, the number of existing multimodal applications, especially mobile applications, is rapidly accelerating. Multimodal Voice Search in particular is a relatively new and compelling use case, and has been implemented in applications by a number of companies, including Google, Microsoft, Yahoo, Vlingo, SpeechCycle, Novauris, AT&T, Openstream, Vocalia, Metaphor Solutions and Sound Hound. Speech offers a welcome means to interact with smaller devices, allowing one-handed and hands-free operation. Users benefit from being able to choose which modalities they find convenient in any situation. The Working Group should be of interest to companies developing smart phones and personal digital assistants or who are interested in providing tools and technology to support the delivery of multimodal services to such devices.
Please note that a related effort has recently been initiated in the W3C by the HTML Speech Incubator Group (HTML Speech XG). The focus of the XG is developing proposals for accessing speech recognition and speech synthesis from HTML5 browsers, and Voice Search and Speech Command Interfaces are possible use cases for these technologies in the browser. However, the XG does not attempt to address modalities other than speech, such as handwriting, emotion, or the wide variety of present and future input modalities. Similarly, it doesn't attempt to address non-browser contexts. In contrast, the Multimodal Architecture provides a generic framework for modality integration and control. Speech in the browser can be seen as a special case of the kind of modality integration covered by the MMI Architecture. The Multimodal Interaction Working Group has been collaboratively working with the XG, and will continue to liaise with them on topics of common interest. For example, the XG has adopted EMMA as a speech recognition result format.

Home appliances, e.g., TV, and home networks

Multimodal interfaces are expected to add value to remote control of home entertainment systems, as well as finding a role for other systems around the home. Companies involved in developing embedded systems and consumer electronics should be interested in W3C's work on multimodal interaction.

Enterprise office applications and devices

Multimodal has benefits for desktops, wall mounted interactive displays, multi-function copiers and other office equipment which offer a richer user experience and the chance to use additional modalities like speech and pens to existing modalities like keyboards and mice. W3C's standardization work in this area should be of interest to companies developing client software and application authoring technologies, and who wish to ensure that the resulting standards live up to their needs.

Intelligent IT ready cars

With the emergence of dashboard integrated high resolution color displays for navigation, communication and entertainment services, W3C's work on open standards for multimodal interaction should be of interest to companies working on developing the next generation of in-car systems.

Medical applications

Mobile healthcare professionals and practitioners of telemedicine will benefit from multimodal standards for interactions with remote patients as well as for collaboration with distant colleagues.

Work to do

Multimodal Architecture and Interfaces (MMI Architecture)

Bring the MMI Architecture to Recommendation: finalize generic architecture and define profiles for using the MMI Architecture with specific use cases

To assist with realizing this goal, the Multimodal Interaction Working Group is tasked with providing a loosely coupled architecture for multimodal user interfaces, which allows for co-resident and distributed implementations, and focuses on the role of markup and scripting, and the use of well defined interfaces between its constituents. The framework is motivated by several basic design goals including (1) Encapsulation, (2) Distribution, (3) Extensibility, (4) Recursiveness and (5) Modularity.

Where practical this should leverage existing W3C work. See the list of dependencies for the information of related groups.

The Working Group may expand the Multimodal Architecture and Interfaces document and include exploration of some language definition which would make it easier to adapt existing I/O devices and software, e.g., an MRCP-enabled ASR engine, to interact with the framework of the life cycle events defined in the architecture.

As the diagram below shows, the Multimodal Architecture includes Modality Components, which process specific modalities such as speech or handwriting, an Interaction Manager, which coordinates processing among the Modality Components, and the Life Cycle events, which support communication between the Interaction Manager and the Modality Components. Modality-specific processing takes place on one or more devices, or remotely. A Life Cycle Event can carry control information or user input. When a Life Cycle Event carries user input as a payload, it uses EMMA, a generic, modality-independent format for representing user input. Within EMMA, the user input can be represented, for example, as EmotionML, InkML, or the semantically interpreted results of speech recognition, depending on the input modality. The Multimodal Architecture also makes use of related standards, as appropriate, such as SRGS, SISR, PLS, and SCXML, developed by the W3C Voice Browser Working Group.

Modality Components Registration & Discovery

The Working Group will investigate and recommend how various Modality Components in the MMI Architecture which are responsible for controlling the various input and output modalities on various devices can register their capabilities and existence with the Interaction Manager. The possible examples of registration information of Modality Components include capabilities of Modality Components such as Languages supported, ink capture and playback, media capture and playback (audio, video, images, sensor data), speech recognition, text to speech, SVG, geolocation and emotion.

It should be noted here that any single modality component in the MMI architecture can potentially encompass multiple physical devices, also more than one modality components could be included in a single device. Given that distinction, the focus of this group is to investigate and document the process for discovery of modality components and their capabilities rather than that for discovery of devices and services, which may have been addressed by other working groups, e.g., the Device APIs Working Group. The Multimodal Interaction Working Group itself will not define any JavaScript APIs for device discovery or registration.

The group will generate a document which describes the detailed definition of the format and process of Registration of Modality Components as well as the process for the Interaction Manager to discover the existence and current capabilities of Modality Components. This document is expected to become a recommendation track document.

Device handling capability

Clear and concrete description on how to handle device interaction within the context of the MMI Architecture is expected. For example device vendors need to know how to integrate the MMI Architecture with existing device handling standards like DLNA and Echonet. So the group will consider to publish guidelines for integrating the MMI Architecture and existing device handling standards. Depending on the expertise of WG participants, the group may prepare a Working Group Note on how to integrate existing device handling standards, e.g., DLNA, and the MMI Architecture. The results could be included in the EMMA 2.0 specification. Also there is a possibility a specific language for CE device handling Modality Component, e.g., Device Handling Markup Language (DHML), will be defined. Please note that there is the Device APIs Working Group which works to standardize client-side APIs that enable the development of Web Applications on browsers and Web Widgets. However, the scope of the Multimodal Interaction Working Group is not only handling device capability within a single client device but also dealing with interaction among independent/multiple devices as Modality Components. The Multimodal Interaction Working Group will liaise with groups that work for device capability including the Device APIs Working Group to discuss what kind of features should be considered not only from MMI viewpoint but also from the other groups' viewpoints.

Extensible Multi-Modal Annotations (EMMA)

EMMA is a data exchange format for the interface between different levels of input processing and interaction management in multimodal and voice-enabled systems. It provides the means for input processing components, such as speech recognizers, to annotate application specific data with information such as confidence scores, time stamps, and input mode classification (e.g. key strokes, touch, speech, or pen). EMMA also provides mechanisms for representing alternative recognition hypotheses including lattice and groups and sequences of inputs. EMMA is a target data format for the semantic interpretation specification developed in the W3C Voice Browser Activity, which defines augmentations to speech grammars that allow extraction of application specific data as a result of speech recognition. EMMA supersedes earlier work on the natural language semantics markup language (NLSML) in the Voice Browser Activity.

EMMA Ver. 1.0 became a W3C Recommendation in February 2009 and it has been implemented by more than 10 different companies and institutions. In the period defined by this charter, the group will actively maintain and address any issues that arise with the existing EMMA 1.0 specification. The group will publish a new EMMA 1.1 version of the specification which incorporates new features that address issues brought up through EMMA implementations:

• Support for a broader range of semantic representation formats
• Revisions to the grammar mechanism to enable inline specification of grammars and specification of both active and matching grammars
• Clarification of medium and mode annotations and extensions to align EMMA with EmotionML

The group will publish a new requirements document for more extensive features that have been requested such as:

• Representation of multimodal outputs for information presentation and embodied conversational agents
• Extended support for multimodal corpora annotation and logging
• Biometric processing

The group may publish an updated 2.0 version of the EMMA specification with the above capabilities.

EmotionML

Bring Emotion Markup Language to Recommendation.

EmotionML provides representations of emotions for technological applications in three different areas: (1) manual annotation of data; (2) automatic recognition of emotion-related states from user behavior; and (3) generation of emotion-related system behavior.

EmotionML has progressed very well in the previous charter. It has received positive feedback at the W3C Workshop on Emotion Markup Language.

As the workshop minutes says, there were the following participants in the workshop:

• Cantoche
• Deutsche Telekom
• DFKI
• Dublin Institute of Technology
• Keio University
• Loquendo
• nViso
• Queens University of Belfast
• Roma Tre University
• Telecom ParisTech
• University of Greenwich

And all of them were interested in adding support for EmotionML in their products and/or research prototypes in the following domains:

• Affective monitoring / emotion detection to verify customer reactions to advertisements;
• Expressive speech synthesis, generating synthetic speech with different emotions;
• Generation of expressive behavior for animated virtual characters displayed on web pages;
• Support for annotating emotions in audio and video resources using crowdsourcing platforms;
• Opinion mining / sentiment analysis from text;
• Modeling and induction of of group emotion in disaster recovery training sessions;

In addition to these plans, EmotionML also has the potential to support applications for people with disabilities, such as educational programs for people with autism and "emotion captioning" for people who are unable to hear emotions expressed by voice or who are unable to see emotions expressed on the face.

In order to assess the potential impact of emotion annotation for the web, the group intends to publish a use cases and requirements document on making emotion annotation usable in web browsers.

Maintenance work

The Working Group will be maintaining the InkML and EMMA specifications; that is, responding to questions and requests on the public mailing list and issuing errata as needed. The Working Group will also consider publishing additional versions of the specifications, depending on such factors as feedback from the user community and any requirements generated for EMMA and InkML by the Multimodal Architecture and Interfaces work and EmotionML.

Success Criteria

For each document to advance to proposed Recommendation, the group will produce a technical report with at least two independent and interoperable implementations for each required feature. The working group anticipates two interoperable implementations for each optional feature but may reduce the criteria for specific optional features.