Multimodal Interaction Use Cases

Abstract

The W3C Multimodal Interaction Activity is developing specifications as a basis for a new breed of Web applications in which you can interact using multiple modes of interaction, for instance, using speech, hand writing, and key presses for input, and spoken prompts, audio and visual displays for output. This document describes several use cases for multimodal interaction and presents them in terms of varying device capabilities and the events needed by each use case to couple different components of a multimodal application.

Status of this Document

This section describes the status of this document at the time of its publication. Other documents may supersede this document. The latest status of this document series is maintained at the W3C.

W3C's Multimodal Interaction Activity is developing specifications for extending the Web to support multiple modes of interaction. This document describes several use cases as the basis for gaining a better understanding of the requirements for multimodal interaction, and the kinds of information flows needed for multimodal applications.

This document has been produced as part of the W3C Multimodal Interaction Activity, following the procedures set out for the W3C Process. The authors of this document are members of the Multimodal Interaction Working Group (W3C Members only). This is a Royalty Free Working Group, as described in W3C's Current Patent Practice NOTE. Working Group participants are required to provide patent disclosures.

Please send comments about this document to the public mailing list: www-multimodal@w3.org (public archives). To subscribe, send an email to <www-multimodal-request@w3.org> with the word subscribe in the subject line (include the word unsubscribe if you want to unsubscribe).

A list of current W3C Recommendations and other technical documents including Working Drafts and Notes can be found at http://www.w3.org/TR/.

1. Introduction

Analysis of use cases provides insight into the requirements for applications likely to require a multimodal infrastructure.

The use cases described below were selected for analysis in order to highlight different requirements resulting from application variations in areas such as device requirements, event handling, network dependencies and methods of user interaction

It should be noted that although the results of this analysis we be used as input to the Multimodal Specification being developed by the W3C Multimodal Interaction Working Group, there is no guarantee that all of these applications will be implementable using the language defined in the specification.

1.1 Use Case Device Classification

Thin Client

A device with little processing power and capabilities that can be used to capture user input (microphone, touch display, stylus, etc) as well as non-user input such as GPS. The device may have a very limited capability to interpret the input, for example a small vocabulary speech recognition, or a character recognizer. The bulk of the processing occurs on the server including natural language processing and dialog management.

An example of such a device may be a mobile phone with DSR capabilities and a visual browser (there could actually be thinner clients than this).

Thick Client

A device with powerful processing capabilities, such that most of the processing can occur locally. Such a device is capable of input capture and interpretation. For example, the device can have a medium vocabulary speech recognizer, a handwriting recognizer, natural language processing and dialog management capabilities. The data itself may still be stored on the server.

An example of such a device may be a recent production PDA or an in-car system.

Medium Client

A device capable of input capture and some degree of interpretation. The processing is distributed in a client/server or a multidevice architecture. For example, a medium client will have the voice recognition capabities to handle small vocabulary command and control tasks but connects to a voice server for more advanced dialog tasks.

1.2 Use Case Summaries

Table 1: Form Filling for air travel reservation

Scenario Details

User wants to make a flight reservation with his mobile device while he is on the way to work. The user initiates the service via means of making a phone call to a multimodal service (telephone metaphore) or by selecting an application (portal environment metaphore). The details are not described here.

As the user moves between networks with very different characteristics, the user is offered the flexibility to interact using the preferred and most appropriate modes for the situation. For example, while sitting in a train, the use of stylus and handwriting can achieve higher accuracy than speech (due to surrounding noise) and protect privacy. When the user is walking, the input and output modalities that more appropriate would be voice with some visual output. Finally, at the office the user can use pen and voice in a synergistic way.

The dialogue between the user and the application is driven by a form-filling paradigm where the user provides input to fields such as "Travel Origin:", "Travel Destination:", "Leaving on date", "Returning on date". As the user selects each field in the application to enter information, the corresponding input constraints are activated to drive the recognition and interpretation of the user input. The capability of providing composite multimodal input is also examined, where input from multiple modalities is combined for the interpretation of the user's intent.

Table 2: Driving Directions

Scenario Details

User wants to go to a specific address from his current location and while driving wants to take a detour to a local restaurant (The user does not know the restaurant address nor the name). The user initiates service via a button on his steering wheel and interacts with the system via the touch screen and speech.

Table 3: Name Dialing

Scenario Details

Janet presses a button on her multimodal phone and says one of the following commands:

• Call Wendy
• Call Wendy on her cell phone
• Call Wendy at work
• Call Wendy Smith at Acme Research

The application initially looks for a match in Janet's personal contact list and if no match is found then proceeds to look in other directories. Directed dialog and tapered help are used to narrow down the search, using aural and visual prompts. Janet is able to respond by pressing buttons, or tapping with a stylus, or by using her voice.

Once a selection has been made, rules defined by Wendy are used to determine how the call should be handled. Janet may see a picture of Wendy along with a personalized message (aural and visual) that Wendy has left for her. Call handling may depend on the time of day, the location and status of the both parties, and the relationship between them. An "ex" might be told to never call again, while Janet might be told that Wendy will be free in half an hour after Wendy's meeting has finished. The call may be automatically directed to Wendy's home, office or mobile phone, or Janet may be invited to leave a message.

2. Use Case Details

2.1 Use-case: Form filling for air travel reservation

Description: The air travel reservation use case describes a scenario in which the user books a flight using a wireless personal mobile device and a combination of input and output modalities.

The device has a touch-enabled display (i.e., supports pen input) and it is voice enabled. The use case describes a rich multimodal interaction model that allows the user to start a session while commuting on the train, continue the interaction while walking to his office and complete the transaction while sat at his office-desk. As the user moves between environments with very different characteristics, the user is given the opportunity to interact using the preferred and most appropriate modes for the situation. For example, while sitting in a train, the use of stylus and handwriting can offer higher accuracy than speech (due to noise) and protect privacy. When the user is walking, the input and output modalities more appropriate would be voice with some visual output. Finally, at the office the user can use pen and voice in a synergistic way.

This example assumes the seamless transition through a variety of connectivity options such as high bandwidth LAN at the office (i.e., 802.11), lower bandwidth while walking (i.e., cellular network such as GPRS) and low bandwidth but in addition intermittent connectivity while on the train (e.g., can get disconnected when going through a tunnel). The scenario also takes advantage of network services such as location and time.

Actors

• User who makes the air travel reservation
• Mobile device with touch-enabled display wireless network connectivity, handwriting recognition capability and limited voice recognition capability on the device.
• Network service with full voice dialog capabilities, connection to travel reservation database and location/time services.

Additional Assumptions

• Data capabilities are available on the communications provider's network. Voice requirements are satisfied either via voice capabilities on the communications provider network or through a DSR framework that utilized the existing data capabilities.
• There are means for describing user and device profile information and means of exchanging this information between server and client.

Table 4: Event Table

2.2 Use-case: Driving Directions

Assumptions

• ASR services are local for simple requests (e.g. session preference setup)
• ASR is server-based for complex requests (e.g. addresses)
• TTS local
• Execution model is hosted on the device.
• single language - with acknowledgement that we will ultimately need language selection
• availability (always on) - with acknowledgement that there may be temporary interruptions due to unexpected circumstances (e.g. tunnels, mountains)
• driver is alone [cannot get assistance]
• Additional applications may be available when the service is initiated via a service selection menu (this is beyond the scope of this use case analysis)
• Initiating recognition requires a single button press. Button press indicating end of speech is optional assuming with preconfigured timeout to stop listening (requiring the user to hold down a button while driving may be dangerous)
• At any time during the session, the user may change display options via the touch screen (includes zooming in and changing route display options). Display options may also be changed using speech by initiating a dialog by pressing the button on the steering wheel

Actors

Primary Device:

• on-board system (in a car) with the following capabilities:

  • graphical display:
    • maps
    • Estimated time of arrival
    • Textual Directions
  • touch screen
  • voice (input and output)
  • keyboard/text input
  • local ASR and TTS processing
  • access to remote servers (ASR and App Server)
  • GPS

Data sources:

• route database
• traffic conditions
• GPS data
• speedometer
• landmarks database and places of interest:
  • nearest gas station
  • nearest restaurant of a specific type
• User Preference Database

Scenario Walkthrough (User point of view)

• Primary Input: Speech
• Secondary Input: Touch Screen
• Speech and Graphical Output
• Preferences are stored on the server to enable multiple users to use the same device (Preferences may be retrieved automatically based on speaker identification or key identification eliminating the need for an authentication dialog)

User wants to go to a specific address from his current location and while driving wants to take a detour to a local restaurant (The user does not know the restaurant address nor the name)

Table 5: Event Table

Protocols:

• HTTP
• Proprietary protocol for connection to ASR server?
• GPS
• Others

Events:

• ASR Events
• Touch Screen Events
• GPS Updates
• Refresh Triggers
• Traffic Alerts
• Others???

Synchronization Issues:

• Spoken Directions must be synchronized with current location
• When route changes while prompts are playing, current prompts must be stopped and new prompts queued. This may be triggered by the following:
  • BSW pressed by user
  • Screen is touched
  • Traffic Update event is received
  • Driver Error
• Screen must be updated to reflect current location and route. This may be triggered by:
  • Refresh Event
  • Change of destination
  • Change of route
  • Driver Error
• Asynchronous events such as traffic updates need to be synchronized with explicit user requests including:
  • Route change requests
  • Display/Output Preference change requests
• Others???

Latency Concerns

• Unanticipated app Server delays may cause directions to be inaccurate

Scenario Considerations

Input Information:

• Starting address/location:
  • explicit street address
  • current location obtained via GPS
  • landmark or place of interest
• Ending address/location:
  • explicit street address
  • landmark or place of interest
• Traffic Conditions
• General preferences:
  • highway vs. scenic route
  • time vs. distance
  • style of output (graphical, turn-by-turn, etc...)
  • units of output (miles vs. kilometers)

Possible Devices:

• Phone with display
• Phone without display (voice only)
• In-dash system (GPS, ASR, TTS)
• PC
• PDA
• Phone (voice + data)
• UMTS

Available Technologies:

• Communication (2.5G, 3G)
• Display (Y/N)
• Application run-time environment (BREW, J2ME, etc)
• Server access

Data sources:

• route database
• traffic conditions
• location [GPS]
• speed and time of arrival [GPS, speedometer]
• landmarks database and places of interest:
  • nearest gas station
  • nearest restaurant of a specific type
• User Preference Database

Output Mechanisms:

• graphical (map)
• text description
• voice
• fax
• dynamic updates (recalculation based on traffic information, driver error, etc...)
• single delivery of results vs. multiple/sequential delivery of results as needed

2.3 Use Case: Multimodal Name Dialling Use Case

Overview

The Name Dialing use case describes a scenario in which users can say a name into their mobile terminals and be connected to the named person based on the called party's availability for that caller.

If the called user is not available, the calling user may be given the choice of either leaving a message on the called user's voicemail system or sending an email to the called user. The called user may provide a personalized message for the caller, including, for example, "Don't ever call me again!"

The called user is given the opportunity of selecting which device the call should be routed to, e.g. work, mobile, home, or voice mail. This may be dependent on the time of day, the called user's location, and the identity of the calling user.

The use case assumes a rich model of name dialling as an example of a premium service exploiting a range of information such as personal and network directories, location, presence, buddy lists and personalization features.

The benefits of making this a multimodal interacton include the ability to view and listen to information about the called user, and to be able to use a keypad or stylus, as an alternative to using voice as part of the name selection process.

Actors

• Caller — user who wishes to place a call

• Called user — user who wishes control over how incoming calls are handled

• Mobile display phone with a lightweight client browser, and optional speaker-dependent minimal speech recognition capabilities

• Network based directory service with speech recognition capabilities, this provides support for looking up names in personal contact lists, as well as in corporate and public directories

• Network based unified messaging service with provision for composing, transferring and playing back messages, including personalized messages intended for specific callers

• User profile database with presence information, buddy lists, and personalized call handling rules

Assumptions

The user has a device with a button that is pushed to place a call. The device has recording capabilities. [voice activation is power hungry and unreliable in noisy environments]

Both voice and data capabilities are available on the communications provider's network (not necessarily as simultaneously active modes).

If the phone supports speech recognition and there is a local copy of the personal phone contact list, then the user's spoken input is first recognized against the local directory for a possible match and if unsuccessful, the request is extended back to the directory provider.

The directory provider has access to a messaging service and to user profiles and presence information. The directory provider thus knows the whereabouts of each registered user - on the phone, at work, unavailable etc.

The directory provider enforces access control rules to ensure individual and corporate privacy. This isn't explored in this use case.

People can be identified by personal names like "Wendy" or by nick names or aliases. The personal contact list provides a means for subscribers to define their own aliases, and to limit the scope of search (there are a lot of Wendy's worldwide).

There is a user agent on the client device with an XHTML browser and optional speaker-dependent speech recognition capabilities.

There is a client server relationship between the user agent on the device and the directory provider.

The dialog could be driven from either the client device or from the network. This doesn't effect the user view, but does alter the events used to coordinate the two systems. This will be explored in a later section.

The Name Dialing use case will be described through the following views:

User view

User pushes a button and says

  "Call Wendy Smith"

It is also possible to say such things as:

  "Call Wendy"

  "Call Wendy Smith at work".

  "Call Wendy at home".

  "Call Wendy Smith on her mobile phone".

Multiple scenarios are possible here:

If local recognition is supported, the utterance will be first processed by a local name dialling application. If there is no match, the recorded utterance is forwarded to a network based name dialling application.

The user's personal contact list will take priority over the corporate and public directories. This is independent of whether the personal list is held locally in the device or in the network.

The following situations can arise when the user says a name:

1. Single match — the caller is presented with information about the callee. This may include a picture taken from the callee's profile. The caller is asked for a confirmation before the call is put through.

2. Multiple matches — if the number of matches is small (perhaps five or less), the caller is asked to choose from the list. This is be presented to the caller via speech and accompanied with a display of a list of names and pictures. The caller can then:

   • Use a button on the phone to select a list item.
   • Point or touch a link on the screen in the presented list.
   • Say index number or expanded name from the presented list.

   A further alternative is to say "that one" as the system speaks each item in the list in sequence. This method is offered in case the user needs hands and eyes free operation, or the device is incapable of displaying the list

3. Lot's of matches, for example, when the caller says a common name. The caller is led through a directed dialog to narrow down the search.

4. No recognition — the recognizer wasn't able to find a match. The user could have failed to say anything, or there could have been too much noise. A tapered help mechanism is invoked. Callers could be asked to repeat themselves, or asked to key in the number or speak it digit by digit.

Assuming that the user successfully makes a selection:

• The system retrieves further information on the called user such as the current location and local time of that user. The information presented may depend on the relationship between the called and calling users. This assumes support for a buddy list and presence capability. The called user may specify her availability for specific individuals or groups of would be callers depending on time of day etc.

• Two scenarios are described here:

  1. The system finds that the called person is currently available. A picture and/or sound bite is provided to the caller. The system places the call and the user is connected to Wendy Smith.

     Post condition: The user is in a call with the intended party.

  2. The system finds that the called person is unavailable. The system attempts to connects to the called user's voicemail system.

     Assuming this succeeds, the system plays the following prompt back to the caller: "Wendy Smith is currently unavailable. She has left this message for you."

     The message is played out. It could be a multimedia message with recorded sound, text, pictures and even short video clips.

     The system plays a prompt back - "Would you like to leave a message?"

     The user says "Yes".

     The user is then connected to the voicemail system and leaves a message for Wendy Smith.

     If Wendy's voicemail box is full or unavailable, the system offers the caller the chance of composing an email. This occupies the caller's storage allocation until it has been sent.

     Post condition: The user has left a message for the intended party.

The availability of the called user may depend on the time of day, whether the called user is away from her work or home location, and who the calling user is. For example, when travelling you may want to take calls on your mobile during the day. Don't you hate it when people call you in the middle of the night because they don't realize what timezone you are in! You may want to make an exception for close friends and family members. There may also be some people whom you never want to accept calls from, not even voice messages!

When a user is notified of an incoming call, the device may present information on the caller including a photograph, name, sound bite, location and local time information, depending on the relationship between the caller and callee. The user then has an opportunity to accept the call or to divert it to voice mail.

Directory provider View

• The client on the user device records the spoken input. The spoken input is recognized against the directory on client device. When this fails, the utterance is extended to the directory provider for recognition.

  If the user device doesn't support local recognition, it may still need to record the utterance, so that the user can start talking immediately without needing to wait for the connection to the directory provider to be completed.

• The directory provider retrieves the profile for the calling user. This has information on which device the user is calling from, the current location of the calling user etc. The calling user is authenticated and authorized.

• The recognizer in the provider recognizes the spoken utterance and returns the result. This result can either be a single entry or a list of possible close matches.

• The server application (hosting the directory provider) now controls the flow of the interaction henceforth.

• The server goes to the database and retrieves more information based on the recognizer result.

• The provider queries the presence of the called user, and personalization information (buddy list, location and presence information, etc.) to construct the content for the response.

• A result may be returned back to the client device in more than one way here:

  A single XHTML page is constructed with both visual picture and audio with the complete name of the recognized match.

  The feedback can include two channels such as visual for the picture and a separate voice channel for playing back the name of the user (an optimization for reduced latency).

• The server creates and transfers a composed page to the client device.

• Once the client receives the content from the application server, multiple scenarios are possible here based on the recognizer result. See user view for details.

• Picking a choice from a list can be done by voice, button or stylus. The user should be able to browse the list, and to revisit the list upon rejecting a confirmation of a preceding choice.

  Example: user says "Call the first one". This utterance is processed by the directory provider to select the first match.

• The directory application may need to apply a directed dialog to narrow the search when there are more than a few matches, or when recognition and tapered help needs to be offered.

What is driving the dialog?

The details of the events depend on whether the dialog is being driven from the network or from the user device.

When the device sends a spoken utterance to the server, the user may have spoken a name such as "Tom Smith" or spoken a command such as "the last one". If the directory search is being driven by the user device, the server's response is likely to be a short list of matches, or a command or error code. To support the application, the server would provide a suite of functions, including the means for the device to set the recognition context, the ability to play specific prompts, and to download information on named users.

If the network is driving the dialog, the device sends the spoken utterance in the same way, but the responses are actions to update the display and local state. If the caller presses a button or uses a stylus to make a selection, this event will be sent to the server. The device and server could exchange low level events, such as a stylus tap at a given coordinate, or higher level events such as which name the user has selected from the list.

Table 6: Event Table

Table 7: Interaction details of displaying and confirming the recognition results

Table 8: No local recognition, all recognition in the Network

3. Acknowledgements

The following people contributed to this document:

• Paulo Baggia, Loquendo
• Art Barstow, Nokia
• Emily Candell, Comverse
• Debbie Dahl, Consultant and Working Group Chair
• Stephen Potter, Microsoft
• Vlad Sejnoha, Scansoft
• Luc Van Tichelin, Scansoft
• Tasos Anastasakos, Motorola
• Lin Chen, Voice Genie
• Jim Larson, Intel Architecture Lab
• T.V. Raman, IBM
• Derek Schwenke, Mitsubishi Electric
• Giovanni Seni, Motorola
• Dave Raggett, W3C/Openwave
• Bennett Marks, Nokia
• Katriina Halonen, Nokia
• Ramalingam Hariharan, Nokia
• Stephane Maes, IBM
• Purush Yeluripati
• Kuansan Wang, Microsoft